FETC 2019 Metrics Messaging Blended Learning julie evans 012919 final
1 like243 views
This document summarizes a presentation about evaluating blended learning implementations. It discusses: 1) Key research from Project Tomorrow on the state of blended learning in K-12 schools, finding it is implemented in many schools and seen to have benefits like increased personalization and engagement. 2) Challenges principals face in implementing blended learning like ensuring student access to technology and developing models of student responsibility. 3) How school administrators currently measure impact, most commonly through teacher and student feedback and achievement results. 4) Details of two efficacy studies Project Tomorrow conducted on blended learning, including factors that affect outcomes and limitations in isolating blended learning impact. Key findings showed maturity in the blended learning model and consistency in
FETC 2019 Metrics Messaging Blended Learning julie evans 012919 final
- 1. Julie A. Evans, Ed.D. Chief Executive Officer, Project Tomorrow @JulieEvans_PT Metrics and Messaging: What Works when Evaluating Blended Learning Implementations Tuesday, January 29, 2019 @ 2 pm
- 2. “Innovating is a process, not an event.” Blended: Using Disruptive Innovation to Improve Schools by Michael B. Horn and Heather Staker
- 3. Today’s discussion ▪ Introduction to Project Tomorrow and the Speak Up Research Project ▪ Key research highlights about blended learning implementations ▪ Metrics for measuring impact – step by step process we recommend ▪ Messaging value and impact to your community ▪ Other resources for you ▪ Your thoughts, comments, ideas! Let’s talk!
- 4. Sign up to get a copy of today’s presentation! 1. Sign up on the table sheets 2. Sign up on the Google form http://bit.ly/2HubYkw
- 5. Getting to know you!
- 6. Tweet with us! Share your thoughts! @JulieEvans_PT @SpeakUpEd @FETC #FETC #blendedlearning
- 7. About Project Tomorrow (www.tomorrow.org) ▪ Nonprofit education organization supporting K-12 education since 1996 ▪ Mission is to ensure all of today’s students are well prepared for the future ▪ Programs and research focus on role of digital tools within the education ecosystem – believe in power of STEAM to support student preparation for college and career success o Speak Up Research Project on Digital Learning: collecting & reporting on the authentic feedback of K-12 stakeholders to inform federal, state & local programs and policies o Efficacy and evaluation studies on impact of digital learning
- 8. About the Speak Up Project (www.tomorrow.org/speakup) ▪ Annual research project since 2003 ▪ Uses online surveys + focus groups ▪ Facilitated 100% through schools and districts ▪ We design online surveys to collect feedback from your K-12 Students, Teachers, Parents, Administrators and Community Members ▪ All K-12 schools – public, private, parochial, charter, virtual - are eligible to participate ▪ Project Tomorrow manages all data collection and reporting for you - 100% free service ▪ Schools get summary report with all locally collected data + state and national data for benchmarks We share national data with federal, state and local policymakers to inform programs and funding
- 9. About the Speak Up Project (www.tomorrow.org/speakup) Standard survey question topics include: ✓ Use of technology to support learning ✓ School climate for innovation ✓ College and career ready skill development/interest ✓ Leadership challenges and capacities ✓ Teachers’ needs for professional learning ✓ How do different stakeholders value digital learning ✓ Emerging trends w/digital tools, content and resources ✓ New classroom models: mobile, blended, flipped ✓ School to home communications ✓ Student safety online & digital citizenship ✓ Designing the ultimate school Since 2003, over 5.4 million K-12 stakeholders have submitted a Speak Up survey
- 10. Key trends from the Speak Up Research • Greater emphasis on linking technology to students’ global skill preparation • Interest, acceptance and implementation of new learning models • Students’ self-directed digital learning influencing expectations for school • Meet the new digital parent – new demands, higher aspirations • Thinking beyond engagement to understand outcomes from digital learning • Increasing criticality for Internet connectivity – at school and at home • Digital learning is a metaphor today for education transformation • The strong connection between leadership and effectiveness
- 11. About our Efficacy & Evaluation Studies Contracted research projects to understand the efficacy of digital learning ▪ Extension of our Speak Up research – 40+ studies since 2007 ▪ Specialization in evaluating mobile learning, new classroom models and the use of digital content within instruction ▪ Study types: feedback studies, efficacy studies, secondary data analysis, evaluations of outcomes, control & treatment groups ▪ Clients include K-12 districts, foundations, companies, nonprofit orgs ▪ Outcomes inform education planning, new initiatives, community engagement strategies, development of new income, understanding product impact, new solution development
- 12. Speak Up Research Findings: State of blended learning in K-12 schools Background and context: research results
- 13. Survey Audience # of Surveys Submitted K-12 Students 340,927 Teachers & Librarians 34,833 Parents 23,159 Administrators 3,249 Community Members 4,611 Speak Up 2017 Data Set: 406,779 respondents About participating schools and districts: 3,641 districts, 10,619 schools 29% urban, 34% rural, 37% suburban, 68% title 1 eligible
- 14. Per the Clayton Christensen Institute (https://www.christenseninstitute.org/), blended learning is a structured education program which includes both traditional learning as well as online learning. In a blended learning model, students spend some time in a traditional classroom with a teacher providing instruction, and some time following an online curriculum (at home or at school) which allows them to have some control over the time, place, path, or pace of their learning. Definition of blended learning used within our research projects:
- 15. 22% 26% 26% 23% 26% 24% 26% 44% 59% 59% 54% 52% 49% 50% 0% 10% 20% 30% 40% 50% 60% 70% ELEMENTARY SCHOOLS MIDDLE SCHOOLS HIGH SCHOOLS URBAN COMMUNITIES SUBURBAN COMMUNITIES RURAL COMMUNITIES TITLE 1 SCHOOLS Blended learning implemented Demonstrated positive academic results from blended learning Where is blended learning being implemented within K-12? Everywhere!
- 16. What do you think are the benefits of implementing blended learning environments in schools?
- 17. What do principals say are the benefits of implementing blended learning? Principals who have implemented blended learning cite these outcomes: 1. Extends learning opportunities for students (66% of the principals agree) 2. Increases personalization of instruction (61%) 3. Increases student engagement in learning (53%) 4. Students are more responsible for their own learning (48%) 5. Enhances the relevancy of the curriculum (48%) 6. Builds stronger school to home communications (46%) 7. Student academic outcomes are improved (45%) 8. Teachers’ skills using technology are improved and technology is used more effectively (40%)
- 18. What do principals say are the challenges of implementing blended learning? Middle school principals identify these as key challenges: ▪ Need to educate parents about at-home responsibilities with blended learning (64%) ▪ How to develop a student culture of responsibility for educational destiny (54%) ▪ Ensuring that every child has access to safe, convenient, sustained and appropriate online access beyond the school campus (49%) ▪ Imperative that schools identify best practices to implement in the classroom (45%) ▪ Also necessary to re-think traditional instructional practices that support blended learning (43%) ▪ Determining how to measure the impact of blended learning on educational goals (42%)
- 19. How are school and district administrators currently evaluating the impact of their digital initiatives?
- 20. How are you measuring the impact of digital learning initiatives?0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Teacher buy- in/feedback Student achievement results Student feedback Class observations Student engagement All District Admin District Admin - small sized districts District Admin - medium sized districts District Admin - large sized districts
- 21. Efficacy and Evaluation Studies: Evaluating impact of blended learning Background and context: research results
- 22. Efficacy and Evaluation Studies: Evaluating impact of blended learning Two different blended learning impact studies designed and conducted by Project Tomorrow: A. Poudre Global Academy (CO) - 2015 o Evaluating impact of blended learning on student achievement B. 9 K-8 parochial schools in Los Angeles – 3 year study – 2019 o Evaluating impact of blended learning on student achievement, enrollment and school financial health
- 23. Efficacy and Evaluation Studies: Evaluating impact of blended learning B. Study: 9 K-8 parochial schools in Los Angeles o Why blended learning? ▪ Desire to improve academic outcomes and close achievement gap – disadvantaged students, students with limited EL ▪ Motivation to change teacher skills using technology ▪ Motivation to update teacher instructional practices ▪ Upgrade profile of school to meet parent expectations ▪ Competitive pressures from charter schools ▪ Goal to better prepare students for demanding high school experiences, college readiness and career opportunities
- 24. Efficacy and Evaluation Studies: Evaluating impact of blended learning B. Study: 9 K-8 parochial schools in Los Angeles o Evaluating impact of blended learning on student achievement ▪ Secondary data analysis ▪ Reading Gr 2-8 ▪ Math Gr 2-8 ▪ Standardized assessments used in all 9 schools ▪ 3 school years of data ▪ Schoolwide and grade level views (aggregated student data) ▪ 2015 – 2018 Analysis theme: to see evidence of effectiveness = consistency and constancy of growth and achievement – schoolwide + at pivotal grades
- 25. In common variables we know about Divergent variables we know about Variables we don’t know about • Grade levels • School culture • Blended learning rotation model • Commitment to closing student achievement gaps • School leadership commitment to blended learning • Provider model • Curriculum products • Implementation years • School environment • Prior emphasis on data or gap analysis • Teacher turnover • Student demographics • Technology platforms • Teacher effectiveness • Principal effectiveness • Teacher skills • Teacher attitudes or values • Fidelity to the model • Parental support or satisfaction • Attendance issues B. Study: 9 parochial schools in Los Angeles – study limitations Result: need to use both traditional metrics and new metrics for the data analysis Result: examine pivotal grades of 3rd and 8th
- 26. B. Study: 9 parochial schools in Los Angeles – key findings – achievement o Maturity with a blended learning model matters. Schools with 3+ years of experience with blended learning had stronger performance in reading and math based upon the review of traditional achievement metrics that emphasis student growth. o Blended learning provides evidence of student proficiency as well as growth. Thinking beyond growth to proficiency measures, tangible outcomes are evident from the blended learning implementations in the following areas: ▪ Sustained increases in students’ scaled scores in reading in grades 3 and 8 ▪ Accomplishment of a minimum of one grade level growth in reading and math in grades 3 and 8 ▪ Eight of nine schools had their 8th graders graduate in spring 2017 with reading levels that are at or with 10% of being classified as college ready
- 27. B. Study: 9 parochial schools in Los Angeles – key findings – achievement o Maturity with a blended learning model matters. 0 10 20 30 40 50 60 70 80 90 Median PR Median SPG % above 65 PR % proficient above CAASPP School 6 School 4 School 8 School 2 School 1 School 3 School 9 School 7 School 5 Schools 5, 6, 7 and 9 each had a minimum of 3 years experience with blended learning
- 28. B. Study: 9 parochial schools in Los Angeles – key findings – achievement o Blended learning provides evidence of student proficiency as well as growth. Grade 3 Reading – Comparative Median Scaled Scores and Annual Growth
- 29. B. Study: 9 parochial schools in Los Angeles – key findings – achievement o Blended learning provides evidence of student proficiency as well as growth. Grade 8 Reading – Comparative Lexile Levels Lexile range for grade 8: 1050 – beginning of 8th grade 1100 – beginning of 9th grade + 1200 – college ready reading
- 30. Today’s discussion ▪ Introduction to Project Tomorrow and the Speak Up Research Project ▪ Key research highlights about blended learning implementations ▪ Metrics for measuring impact – step by step process we recommend ▪ Messaging value and impact to your community ▪ Other resources for you ▪ Your thoughts, comments, ideas! Let’s talk!
- 31. Project Tomorrow’s recommendations for a step-by-step process for evaluating the impact of your blended learning programs 1. Identify your goal(s) for blended learning 2. Determine the value of those goal(s) 3. Establish the necessary proof cases 4. Verify realism of measuring outcomes 5. Agree on logistics of measurement and reporting 6. Determine most appropriate messaging for outcomes
- 32. Project Tomorrow’s recommendations for a step-by-step process for evaluating the impact of your blended learning programs 1. Identify your goal(s) for blended learning ▪ Back to the beginning – what are you trying to achieve? ▪ Does blended learning as an initiative fit that goal? ▪ Is there some modicum of anticipated success? ▪ Be careful about “goal bingeing” ▪ “One goal is a goal, two goals are a goal and a half, three goals = no plan” Common driving goal for all 9 LA schools: closing achievement gap through increased academic achievement for all students
- 33. Project Tomorrow’s recommendations for a step-by-step process for evaluating the impact of your blended learning programs 2. Determine the value of those goal(s) ▪ What does success achieving this goal mean to your district, your school & to you? ▪ How does achieving this goal support your organizational vision or mission? ▪ Does the VOI (value of investment) align with the effort? ▪ Do we have stakeholder buy-in on the value of this goal? ▪ What happens if we don’t do this …. What is the impact of not achieving this goal? Common value proposition: proving academic achievement with blended learning will result in increased enrollments and financial stability
- 34. Project Tomorrow’s recommendations for a step-by-step process for evaluating the impact of your blended learning programs 3. Establish the necessary proof cases ▪ What is the physical representation of success with this goal? ▪ What constitutes adequate proof of success with this goal? ▪ What types of proof are necessary – ex: quantitative vs. qualitative? ▪ Is there stakeholder agreement as to what constitute adequate proof? ▪ Does the proof case carry the same weight with different audiences? ▪ Do your audiences need an education as to why this proof is valuable? Common proof case: what matters to parents are test score results, acceptance into prestigious high schools, engaged happy children, teachers that care about their child
- 35. Project Tomorrow’s recommendations for a step-by-step process for evaluating the impact of your blended learning programs 4. Verify realism of measuring outcomes ▪ Is it realistic and possible to measure outcomes to support that proof case? ▪ Is it realistic and possible to collect supporting data for the proof case? ▪ Will the measured outcomes be valid? ▪ Will the measured outcomes be acceptable or adequate to pass muster with the key stakeholders? Realistic and possible: secondary data analysis of existing achievement data Not realistic or possible (at this time): measuring levels of engagement or teacher concern for children
- 36. Project Tomorrow’s recommendations for a step-by-step process for evaluating the impact of your blended learning programs 5. Agree on logistics of measurement and reporting ▪ Agreement is essential here: are we all on the same page? ▪ Big questions to ask: what, where, when, who ▪ Who should do the data collection and analysis to establish validity? ▪ Who should create the reporting to ensure validity? ▪ Are the analytical and reporting processes appropriate for the task? Agreement: Buy-in from principals as to the data analysis process and sharing data with the third party evaluator; value of the third party from a reporting perspective
- 37. Project Tomorrow’s recommendations for a step-by-step process for evaluating the impact of your blended learning programs 6. Determine most appropriate messaging for outcomes ▪ Back to the beginning: what are you trying to achieve? ▪ Does the evaluation findings support those goals? Or are there surprises? ▪ What do your stakeholders need to understand the value of the process – a lot of data, analyzed information, next steps …. ? ▪ How to articulate your messaging so that it achieves your goals? ▪ Solicit feedback on your messaging and monitor reactions Messaging outcomes: Selecting high impact statements that resonate with targeted audiences (ex: 8th graders reading at college ready levels)
- 38. Project Tomorrow’s recommendations for a step-by-step process for evaluating the impact of . . . . . any of your digital learning initiatives 1. Identify your goal(s) for blended learning 2. Determine the value of those goal(s) 3. Establish the necessary proof cases 4. Verify realism of measuring outcomes 5. Agree on logistics of measurement and reporting 6. Determine most appropriate messaging for outcomes
- 39. Today’s discussion ▪ Introduction to Project Tomorrow and the Speak Up Research Project ▪ Key research highlights about blended learning implementations ▪ Metrics for measuring impact – step by step process we recommend ▪ Messaging value and impact to your community ▪ Other resources for you ▪ Your thoughts, comments, ideas! Let’s talk!
- 40. What conditions or elements have to exist to make a blended learning implementation successful?
- 41. ❖ Supportive, visionary, involved leadership ❖ Teacher confidence and comfort with tools and process ❖ Continuity of staff as model matures ❖ Appropriate tools ❖ Culture that values data and improvement ❖ Classroom practices and habits that work ❖ Student growth mindset: empowerment around learning ❖ Parents that understand and value personalized learning ❖ Maturity with model: move from growth to proficiency ❖ Be open to revisions, re-invention, re-imagining of your initiative as it matures From Project Tomorrow research: what makes blended learning successful? These 10 elements are essential:
- 42. More resources available at www.tomorrow.org National Speak Up reports and infographics Targeted and thematic reports Educational Equity Digital learning trends School communications Mobile learning Blended learning outcomes Presentations, podcasts and webinars Services: consulting, workshops, evaluation and efficacy studies ✓ We have expertise in the evaluation of mobile implementations, new classroom models and digital content usage
- 43. Invitation for your school or district to participate in Speak Up 2018 Online surveys for: o K-12 students – individual + group o Parents – English and Spanish o Teachers o Librarians/Media Specialists o School Site & District Administrators o Technology Leaders o Community Members Surveys open now through June 28, 2019 Learn more www.tomorrow.org/speakup 100% free service to all schools and districts – including reports with comparison data
- 44. Sign up to get a copy of today’s presentation! 1. Sign up on the table sheets 2. Sign up on the Google form http://bit.ly/2HubYkw
- 46. Thank you for joining me today! Please review this session on the mobile app. Julie A. Evans, Ed.D. Chief Executive Officer, Project Tomorrow @JulieEvans_PT jevans@tomorrow.org Metrics and Messaging: What Works when Evaluating Blended Learning Implementations 1/29/19 @ 2 pm – C142