This is Hexagon Tool Analysis Overview.pdf
- 1. National Implementation Research Network @impscience Selecting and Operationalizing Effective Practices Facilitated by: Sophia Farmer Correy Watkins
- 2. National Implementation Research Network @impscience Learning Objectives By the end of this session participants will be able to . . . ● General Overview ● Use the Hexagon tool to select practices/programs 2
- 3. National Implementation Research Network @impscience Selecting Practices
- 4. In your experience, how are effective practices typically chosen?
- 5. National Implementation Research Network @impscience Exploration: Selection of an Evidence Based Practice Use Data to Identify Needs of Target Population Look for “best Evidence” to address need Assess Fit, Readiness or Usability, Capacity 5
- 6. National Implementation Research Network @impscience Which step do you see as skipped most often? 1) Use Data to Identify Needs of Target Population 2) Look for “best Evidence” to address need 3) Assess Fit, Readiness or Usability, Capacity 6 1 2 3
- 7. National Implementation Research Network @impscience Assessing Need and Contextual Fit ● Involve diverse stakeholder engagement ● Uses multiple methods and data sources ● Improves implementation and sustainability potential 7
- 8. National Implementation Research Network @impscience Hexagon Tool 8
- 9. National Implementation Research Network @impscience The Hexagon Tool ● Developed for use in implementation informed efforts to identify contextual fit of sites and programs or practices ● For use by organizations and communities ● Reviewed and edited by the Racial and Ethnic Equity and Inclusion Team (REEI) Adapted from Blase, K., Kiser, L., and Van Dyke, M. (2013). 9 How many have used the Hexagon Tool?
- 10. National Implementation Research Network @impscience The Hexagon Tool: Site Characteristic of Need 10 Does the practice: • Meet the needs of the identified focus population and subpopulations? • Address service or system gaps? • Align with community perception of needs and assets?
- 11. National Implementation Research Network @impscience The Hexagon Tool: Site Characteristic of Fit 11 Does the implementing site: • Identify the practice as fitting with community and family values, culture, and history? • See alignment with other priorities? • Identify alignment with community, regional, and state priorities?
- 12. National Implementation Research Network @impscience The Hexagon Tool: Site Characteristic of Capacity 12 Can the implementing site: • Meet minimum staff qualifications? • Ensure alignment with other priorities?
- 13. National Implementation Research Network @impscience The Hexagon Tool: Practice Characteristic of Evidence 13 Does the practice: • Offer evidence of outcomes, for whom and in what conditions? • Provide outcome, fidelity, and cost effectiveness data?
- 14. National Implementation Research Network @impscience The Hexagon Tool: Practice Characteristic of Usability 14 Does the practice: • Have a well-defined purpose? • Offer mature sites to observe? • Share adaptations for context and population?
- 15. National Implementation Research Network @impscience The Hexagon Tool: Practice Characteristic of Supports 15 Does the practice: • Offer expert assistance? • Offer external resources for implementing sites? • Provide racially equitable impact assessment?
- 16. National Implementation Research Network @impscience • Develop a shared understanding of the need to be addressed, population to be served, and practice • Identify program(s) or practice(s) to examine • Review indicators to develop a plan to have needed data accessible during discussion • Prioritize components (if appropriate) • Is team inclusive of all voices needed? 1 The Hexagon Tool: How do you Use it?
- 17. National Implementation Research Network @impscience • Review and discuss each indicator • Make notes as needed • Team rates component using 5- point Likert scale 2 • Develop a shared understanding of the need to be addressed, population to be served, and practice • Identify program(s) or practice(s) to examine • Review indicators to develop a plan to have needed data accessible during discussion • Prioritize components (if appropriate) • Is team inclusive of all voices needed? 1 The Hexagon Tool: How do you Use it?
- 18. National Implementation Research Network @impscience • Using discussion, notes, and ratings • Make recommendation to: • Adopt • Replicate • De-implement 3 • Review and discuss each indicator • Make notes as needed • Team rates component using 5- point Likert scale 2 • Identify program(s) or practice(s) to examine • Develop a shared understanding of the need to be addressed, population to be served, and practice • Review indicators to develop a plan to have needed data accessible during discussion • Prioritize components (if appropriate) • Is team inclusive of all voices needed? 1 The Hexagon Tool: How do you Use it?
- 19. National Implementation Research Network @impscience Activity Discussion Which factor(s) do you see considered most frequently in selection? Why do you think that is? 19
- 20. National Implementation Research Network @impscience The Hexagon Tool in Use 20
- 21. National Implementation Research Network @impscience Graduation Rate Case Study Needs Assessment: ● Decreased Graduation Rates ● Decreased Parent Engagement ● Decreased Student Connection 21
- 22. National Implementation Research Network @impscience Five Whys Our graduation rates are declining 22 Our graduation rates are declining. Students are not attending school. Students do not feel safe or supported. There are too many fights and drugs in the school. Students do not trust or feel connected to staff to reach out for help. Students do not trust or feel connected to staff to reach out for help. students do not trust or feel connected to staff. because…
- 23. National Implementation Research Network @impscience Mentoring & Mentoring A trained mentor is assigned to each student. The mentor: ● Monitors academic performance and behavior ● Provides support through mentoring, counseling, and advocacy ● Connects student and family to needed resources and services ● Communicates frequently with teachers and other providers ● The mentor meets with assigned students at least twice monthly or more frequently as needed during school hours 23
- 24. National Implementation Research Network @impscience Parallel Thinking with the Hexagon Tool ● Use the Hexagon Tool alongside this team. ● After hearing data and information shared, how would you rate the M&M practice in each area? 24
- 25. National Implementation Research Network @impscience Need Population ○ 35.4% white, 35% Black and African American, 17% Hispanic or Latino, 5% Asian, 3% American Indian or Alaskan Native, 3.8% Two or more races, .1% Hawaiian or pacific islander ○ 60% meet criteria for free and reduced lunch ○ 20% are English Learners ○ 17% receive special education services Data ○ Graduation rates declining (78% to 65% in 3 yrs) ○ Attendance rates declining (88% to 78% in 3 yrs) ○ 72% to 65% of parents agreed school is safe and caring ○ 68% to 58% of students agreed school is safe and caring ○ Focus group data with students and parents: Lack of connection to staff; too many fights; mental illness and drug use rates on the rise 25
- 26. National Implementation Research Network @impscience Need 5-Strongly Meets Need The implementing site has demonstrated a comprehensive understanding of how the program or practice meets the needs of the focus population. The implementing site has included three or more data sources when conducting the needs assessment including administrative data and perspectives of staff, community partners and children, youth and families; and has disaggregated data to identify needs of specific and relevant subpopulations. 4-Meets Need The implementing site has demonstrated an understanding of how the program or practice meets the needs of the focus population. The implementing site has included two or more data sources when conducting the needs assessment including administrative data and perspectives of staff, community partners and children, youth and families; and has disaggregated data to identify needs of specific and relevant sub populations. 3-Somewhat Meets Need The implementing site has demonstrated some understanding of how the program or practice meets the needs of the focus population. The implementing site has included two or more data sources when conducting the needs assessment including administrative data and perspectives of staff, community partners and children, youth and families; but has not disaggregated these data. 2-Minimally Meets Need The implementing site has demonstrated minimal understanding of how the program or practice meets the needs of the focus population. The implementing site has included only administrative when conducting the needs assessment, and has not disaggregated these data. 1-Does Not Meet Need The implementing site has not demonstrated an understanding of how the program or practice meets the needs of the focus population. 26
- 27. National Implementation Research Network @impscience Evidence Two randomized control trials with similar demographics to current population: • Significant positive effect on staying in school • Potentially positive effects for progressing in school One of the randomized control studies: • Students with emotional or behavior disabilities receiving M&M services were statistically significantly less likely to have dropped out than students with emotional or behavioral disabilities who did not receive M&M services 27
- 28. National Implementation Research Network @impscience What is your rating for Evidence? 5-High Evidence The program or practice has documented evidence of effectiveness based on at least two rigorous, external research studies with the focus population and control groups, and has demonstrated sustained effects at least on year post treatment. 4-Evidence The program or practice has demonstrated effectiveness with one rigorous, external research study with the focus population and a control group. 3-Some Evidence The program or practice shows some evidence of effectiveness through less rigorous research studies with the focus population and comparison groups. 2-Minimal Evidence The program or practice is guided by a well-developed theory of change or logic model for the focus population and has demonstrated a relationship between the program or practice and outcomes based on an evaluation or practice-based evidence. 1-No Evidence The program or practice does not have a well-developed theory of change or logic model and has not demonstrated a relationship between the program or practice and outcomes based on an evaluation or practice-based evidence. 28
- 29. National Implementation Research Network @impscience Evidence Ratings from Scenario’s Team 5-High Evidence The program or practice has documented evidence of effectiveness based on at least two rigorous, external research studies with the focus population and control groups, and has demonstrated sustained effects at least on year post treatment. 4-Evidence The program or practice has demonstrated effectiveness with one rigorous, external research study with the focus population and a control group. 3-Some Evidence The program or practice shows some evidence of effectiveness through less rigorous research studies with the focus population and comparison groups. 2-Minimal Evidence The program or practice is guided by a well-developed theory of change or logic model for the focus population and has demonstrated a relationship between the program or practice and outcomes based on an evaluation or practice-based evidence. 1-No Evidence The program or practice does not have a well-developed theory of change or logic model and has not demonstrated a relationship between the program or practice and outcomes based on an evaluation or practice-based evidence. 29
- 30. National Implementation Research Network @impscience Fit Priorities • Goals within district strategic plan for increasing graduation rates and creating safe and caring environments • SEA has improving graduations rates within their state strategic plan overall Values • Focus group data revealed parents and community members expressed need for students to feel safe and supported Existing Initiatives • Several counseling programs in use to teach effective coping and anger management skills • Early Warning Intervention System just installed, staff need training • 2 community coalitions exist to support vocational training and support graduation 30
- 31. National Implementation Research Network @impscience 5-Strong Fit The program or practice fits with all of the priorities of the implementing site and community values, including the values of racially, ethnically, culturally, and linguistically specific population; and other existing initiatives. 4-Fit The program or practice fits with all of the priorities of the implementing site and community values; however, the values of racially, ethnically, culturally, and linguistically specific populations and alignment with other initiatives have not been assessed for fit. 3-Some Fit The program or practice fits with all of the priorities of the implementing site, but it is unclear whether it aligns with community values and other existing initiatives. 2-Minimal Fit The program or practice fits with some of the priorities of the implementing site, but it is unclear whether it aligns with community values and other existing initiatives. 1-Does Not Fit The program or practice does not fit with the priorities of the implementing site, community values, or other existing initiatives. 31 What is your rating for Fit?
- 32. National Implementation Research Network @impscience Fit Rating from Scenario’s Team 5-Strong Fit The program or practice fits with all of the priorities of the implementing site and community values, including the values of racially, ethnically, culturally, and linguistically specific population; and other existing initiatives. 4-Fit The program or practice fits with all of the priorities of the implementing site and community values; however, the values of racially, ethnically, culturally, and linguistically specific populations and alignment with other initiatives have not been assessed for fit. 3-Some Fit The program or practice fits with all of the priorities of the implementing site, but it is unclear whether it aligns with community values and other existing initiatives. 2-Minimal Fit The program or practice fits with some of the priorities of the implementing site, but it is unclear whether it aligns with community values and other existing initiatives. 1-Does Not Fit The program or practice does not fit with the priorities of the implementing site, community values, or other existing initiatives. 32
- 33. National Implementation Research Network @impscience Usability Operationalized Principles • Formulated Set of Beliefs and Principles Core Components • Identified components that describe actions of the mentor and program coordinator • Fidelity assessment-don’t know Adaptations • Limited guidance on adaptation to different contexts • Has been used with various cultural populations (e.g., African Americans, Hispanic or Latino, and Native American or Alaskan Indian) and in different settings (urban and rural) within the state with some success 33
- 34. National Implementation Research Network @impscience 5-Highly Usable The program or practice has operationalized principles and core components that are measurable and observable and has a validated fidelity assessment; modifiable components are identified to support contextualization for new settings or populations. 4-Usable The program or practice has operationalized principles and core components that are measurable and observable and has tools and resources to monitor fidelity but does not have a validated fidelity measure; modifiable components are identified to support contextualization for new settings or focus populations. 3-Somewhat Usable The program or practice has operationalized principles and core components that are measurable and observable but does not have a fidelity assessment; modifiable components are not identified. 2-Minimally Usable The program or practice has identified principles and core components; however, the principles and core components are not defined in measurable and observable terms; modifiable components are not identified. 1-Not Usable The program or practice does not identify principles and core components. 34 What is your rating for Usability? .
- 35. National Implementation Research Network @impscience Usability Rating from Scenario’s Team 5-Highly Usable The program or practice has operationalized principles and core components that are measurable and observable and has a validated fidelity assessment; modifiable components are identified to support contextualization for new settings or populations. 4-Usable The program or practice has operationalized principles and core components that are measurable and observable and has tools and resources to monitor fidelity but does not have a validated fidelity measure; modifiable components are identified to support contextualization for new settings or focus populations. 3-Somewhat Usable The program or practice has operationalized principles and core components that are measurable and observable but does not have a fidelity assessment; modifiable components are not identified. 2-Minimally Usable The program or practice has identified principles and core components; however, the principles and core components are not defined in measurable and observable terms; modifiable components are not identified. 1-Not Usable The program or practice does not identify principles and core components. 35
- 36. National Implementation Research Network @impscience Capacity Workforce • 5 mentors and program coordinator need to be selected, job descriptions adjusted, time for role • BA in ED, Psych, Social work is needed • 2 staff currently available to serve as mentors • Counselor meets qualifications for coordinator • Current staff do not represent the culture or language of students Financial Supports • Some Title funds and State Aid funds to support training activities • Additional funds needed to support hiring and changes to data system Organization Supports • Several school procedures need to be refined to support schedule • Principal and Dean of Students participating in selection and implementation team • Current data system and Early Warning Intervention system need to be altered • Staff need training for the data system and Early Warning Intervention System 36
- 37. National Implementation Research Network @impscience 5-Strong Capacity The implementing site adopting this program or practice has all of the capacity necessary, including all of the following: a qualified workforce, financial supports, technology supports and administrative supports required to implement and sustain the program or practice with integrity. 4-Adequate Capacity The implementing site adopting this program or practice has most of the capacity necessary, including three of the following: a qualified workforce, financial supports, technology supports and administrative supports required to implement and sustain the program or practice with integrity. 3-Some Capacity The implementing site adopting this program or practice has some of the capacity necessary, including two of the following: a qualified workforce, financial supports, technology supports and administrative supports required to implement and sustain the program or practice with integrity. 2-Minimal Capacity The implementing site adopting this program or practice has minimal capacity necessary, including only one of the following: a qualified workforce, financial supports, technology supports and administrative supports required to implement and sustain the program or practice with integrity. 1-No Capacity The implementing site adopting this program or practice does not have the capacity necessary, including any of the following: a qualified workforce, financial supports, technology supports and administrative supports required to implement and sustain the program or practice with integrity. 37 What is your rating for Capacity? .
- 38. National Implementation Research Network @impscience Capacity Rating from Scenario’s Team 5-Strong Capacity The implementing site adopting this program or practice has all of the capacity necessary, including all of the following: a qualified workforce, financial supports, technology supports and administrative supports required to implement and sustain the program or practice with integrity. 4-Adequate Capacity The implementing site adopting this program or practice has most of the capacity necessary, including three of the following: a qualified workforce, financial supports, technology supports and administrative supports required to implement and sustain the program or practice with integrity. 3-Some Capacity The implementing site adopting this program or practice has some of the capacity necessary, including two of the following: a qualified workforce, financial supports, technology supports and administrative supports required to implement and sustain the program or practice with integrity. 2-Minimal Capacity The implementing site adopting this program or practice has minimal capacity necessary, including only one of the following: a qualified workforce, financial supports, technology supports and administrative supports required to implement and sustain the program or practice with integrity. 1-No Capacity The implementing site adopting this program or practice does not have the capacity necessary, including any of the following: a qualified workforce, financial supports, technology supports and administrative supports required to implement and sustain the program or practice with integrity. 38
- 39. National Implementation Research Network @impscience Supports Publisher ○ 2-day training for administrators and teams ○ 2-day training for “mentors” ○ 1-day booster session ○ Coaching supports are not available ○ Cost of trainings: $300 per person ○ Program manual available with theoretical foundation, components, and process for implementing. Includes monitoring form, goal setting guidelines and sample promotional materials. ○ Fidelity measure is not available SEA ○ 2-day training for administrators and teams ○ Three coaching observations of mentors and debrief sessions of observation results with mentors ○ No cost for supports, must qualify for services based on low school performance 39
- 40. National Implementation Research Network @impscience 5-Well Supported Comprehensive resources are available from an expert (a program developer or intermediary) to support implementation, including resources for building the competency of staff (staff selection, training, coaching, fidelity) and organizational practice (data system and data use support, policies and procedures, stakeholder and partner engagement). 4-Supported Some resources are available to support implementation, including limited resources to support staff competency (e.g., training, coaching) and limited resources to support organizational changes (e.g., data systems). 3-Somewhat Supported Some resources are available to support competency development or organizational development, but not both. 2-Minimally Supported Limited resources are available beyond a curriculum or one time training. 1-Not Usable Few to no resources are available to support implementation. 40 What is your rating for Supports? Enter in chat.
- 41. National Implementation Research Network @impscience Supports Rating from Scenario’s Team 5-Well Supported Comprehensive resources are available from an expert (a program developer or intermediary) to support implementation, including resources for building the competency of staff (staff selection, training, coaching, fidelity) and organizational practice (data system and data use support, policies and procedures, stakeholder and partner engagement). 4-Supported Some resources are available to support implementation, including limited resources to support staff competency (e.g., training, coaching) and limited resources to support organizational changes (e.g., data systems). 3-Somewhat Supported Some resources are available to support competency development or organizational development, but not both. 2-Minimally Supported Limited resources are available beyond a curriculum or one time training. 1-Not Usable Few to no resources are available to support implementation. 41
- 42. National Implementation Research Network @impscience Activity Scenario Overall Ratings Evidence 4 Supports 3 Usability 4 Need 4 Fit 4 Capacity 3 42
- 43. National Implementation Research Network @impscience Activity Capturing Your Thoughts ● What are your overall ratings? ● Would you recommend the M & M Program? ● Share out learnings and thoughts about your experiences using the Hexagon Tool Evidence Supports Usability Need Fit Capacity 43
- 44. National Implementation Research Network @impscience Benefits of Engaging in “Hexagon” Process ● Engages leadership in decision making ● Promotes buy-in and creates readiness ● Provides talking points in communication (e.g., rationales for why this practice and not another practice) ● Provides data and information to guide planning necessary infrastructure supports (e.g., installation stage activities) 44
- 45. National Implementation Research Network @impscience Selection Process Using the Hexagon Tool 45
- 46. National Implementation Research Network @impscience Hexagon Tool Process Before ● Identify Team or Group ● Identify Need ● Identify Practice Options ● Gather Needed Information During ● Review & Discuss Each Indicator ● Make Ratings Using Scale ● Determine Adopt, Replicate, or De- Implement After ● Communicate and Use ● Information to Inform ● Implementation Planning 46
- 47. National Implementation Research Network @impscience Activity Reflect on Tips and Traps ● Are there additional ones you have seen in your process? ● What strategies have you identified for addressing the traps? 47
- 48. National Implementation Research Network @impscience Learning Objectives How did we do? By the end of this session, participants will be able to . . . ● General Overview ● Use the Hexagon tool to select practices/programs 48
- 49. National Implementation Research Network @impscience Visit NIRN @impscience @impscience https://nirn.fpg.unc.edu/ More information https://nirn.fpg.unc.edu/ai- lessons-and-short-courses View Interactive Lesson 1 for more detailed information of the Hexagon Process. 49
- 50. National Implementation Research Network @impscience This document is based on the work of the National Implementation Research Network (NIRN). © 2021 NIRN-UNC This content is licensed under Creative Commons license CC BY-NC-ND, Attribution-NonCommercial- NoDerivs . You are free to share, copy, distribute and transmit the work under the following conditions: Attribution — You must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work); Noncommercial — You may not use this work for commercial purposes; No Derivative Works — You may not alter, transform, or build upon this work. Any of the above conditions can be waived if you get permission from the copyright holder. The mission of the National Implementation Research Network (NIRN) is to contribute to the best practices and science of implementation, organization change, and system reinvention to improve outcomes across the spectrum of human services. email: nirn@unc.edu Copyright 50
- 51. National Implementation Research Network @impscience 51