NCCMT Spotlight Webinar: Program Evaluation Toolkit

Editor's Notes

• #9: the NCCPH program is dispersed across the country with 6 National Collaborating Centres the National Collaborating Centre for Methods and Tools is located at McMaster University, in Hamilton 4 of the other NCC’s support the use of research evidence in specific public health content areas NCCMT and NCC Healthy Public Policy work across content areas the focus of NCCMT improving access to, and use of, methods and tools that support moving research evidence into decisions related to public health practice, programs, and policyin Canada.
• #10: NCCMT offers a products and services to help apply research evidence in decision making This presentation today is going to provide an overview of the Online Learning Opportunities that NCCMT offers.
• #12: Kyle has worked at the Centre for the past five years now. In his time at the Centre, he has helped in the development and roll-out of the Centre’s Measures Database and currently oversees the on-going updates to the database. His additional roles at the Centre have included evaluation support (including webinars on selecting measures and indicators, and communicating evaluation findings), helping build the Centre’s internal evaluation and performance measurement capacity, serving as the chair of the Centre’s Organizational Health and Wellness Committee, and researching effective programs and practices as part of the Centre’s Evidence In-sight program.
• #13:  Melissa Jennings has been with the Centre for four years. Her first role was to oversee the evaluation support program which included funding, consultative supports and resources directed to program evaluation activities in child and youth mental health agencies. More recently she has been supporting a range of activities, including program evaluation in lead and core service agencies announced as part of the Ontario government’s Moving on Mental Health plan.
• #14: Thanks for that introduction, so let’s get started on talking about our program evaluation toolkit
• #16: So the first thing I want to go over is how to find the ETK. If you go to the Centre’s website www.excellenceforchildandyouth.ca, you’ll see a tab for the Centre’s resource hub. <next slide>
• #17: That will take you to this picture where you can find all kinds of different Centre resources. We’ll talk a little more later on about some of these resources that complement the toolkit but for now we’ll click on the toolkit button down in the bottom right corner
• #18: You’ll scroll through some of the different And here you’ll see all the toolkits the Centre has to offer across a range of topics like Youth Engagement, Youth Suicide prevention and down in the Support services section is where you’ll find the ETK
• #19: A quick note I want to mention is that the ETK as well as all of the centre products mentioned today are also available in French. I know there are a few people here today who might like to know so I just wanted to bring that up before we dive into the toolkit
• #20: So this is the toolkit
• #21: And it’s split into three main sections: planning, using and doing and each section has details information about steps or, rather, aspects that are relevant to each of those three stages. The planning section covers a wide variety of topics
• #22: And it’s split into three main sections: planning, using and doing and each section has details information about steps or, rather, aspects that are relevant to each of those three stages
• #23: And it’s split into three main sections: planning, using and doing and each section has details information about steps or, rather, aspects that are relevant to each of those three stages
• #24: The Centre’s vision with each of these is to meet agencies where they are at in the process. When we provide evaluation support we like to meet agencies where they’re at, rather they’re just starting out or a it further along in the process and ready to start doing an evaluation. The idea is we would meet people where they are along the process and tailor our supports to meet you where you’re at and move you forward
• #25: So one thing you’ll notice throughout the guide are these “download example” or “download worksheet” links. And what these are is they’re a document that you can download directly through the toolkit that applies directly to the section
• #26: So one thing you’ll notice throughout the guide are these “download example” or “download worksheet” links. And what these are is they’re a document that you can download directly through the toolkit that applies directly to the section
• #28: So, as we mentioned, the guide does focus quite heavily on the planning side of the evaluation process. This was where the majority of the agencies we work with were starting from so the guide really focuses a lot on those who are starting right from step one. And we’ll now spend a bit of time diving into
• #29: So the planning section kicks off with stakeholder engagement and talks a bit about some of the reasons you would want to involve various stakeholders. Some examples of who could be a stakeholder are included, so these could be anyone from funders, to program staff, to program participants, to collaborating agencies or community leaders. One provided attachment in this section covers identifying relevant stakeholders and helps work you through identifying who these may be for your evaluation and some of the key questions to ask yourself about each group such as what does this stakeholder bring to the eval, how important their perspective is and what may be some of their motivation to participate
• #30: One provided attachment in this section covers identifying relevant stakeholders and helps work you through identifying who these may be for your evaluation and some of the key questions to ask yourself about each group such as what does this stakeholder bring to the eval, how important their perspective is and what may be some of their motivation to participate
• #31: The next section covers an area we work quite extensively with people in the planning phase and that is developing a program logic model and this section features a lot of attachments to help walk people through the development of a logic model. So this includes things like a glossary to go over some common terms used with a logic model
• #32: The next section covers an area we work quite extensively with people in the planning phase and that is developing a program logic model and this section features a lot of attachments to help walk people through the development of a logic model. So this includes things like a glossary to go over some common terms used with a logic model
• #33: The next section covers an area we work quite extensively with people in the planning phase and that is developing a program logic model and this section features a lot of attachments to help walk people through the development of a logic model. So this includes things like a glossary to go over some common terms used with a logic model
• #34: The next section covers an area we work quite extensively with people in the planning phase and that is developing a program logic model and this section features a lot of attachments to help walk people through the development of a logic model. So this includes things like a glossary to go over some common terms used with a logic model
• #35: The next section I wanted to highlight involves actually building your evaluation framework. This piece follows a whole section related to identifying indicators and talking about different kinds of outcomes you might be trying to measure
• #36: A quick summary of some practical points you may want to consider, but what I want to draw your attention to is this factsheet we’ve been granted permission to distribute which covers in great detail some aspects related to measurement specifically.
• #37: The next resource I want to highlight are these templates for developing an evaluation framework. So this is the template, and there’s one for process evaluations and one for outcome evaluations. The two templates are exactly they same they just help to focus attention on different aspects and really help you to outline and put things in concrete terms
• #38: Examples of differences between process and outcome eval questions
• #39: Using the example provided with our suicide prevention program before here’s how it could look filled in
• #40: Using the example provided with our suicide prevention program before here’s how it could look filled in
• #42: So I want to take us back to the learning hub to go over some of those additional resources we mentioned earlier that help complement the evaluation toolkit <next slide>
• #43: The first thing I want to draw your attention to is the learning modules over here and once you click there and scroll down
• #44: You’ll see there are six modules all dedicated to evaluation and the first thing you’ll notice is that we have entire modules dedicated to each planning, doing and using evaluation. They cover content similar to what we’ve already talked about in the toolkit so I want to talk a little bit instead about some of the other evaluation modules
• #45: So this is the module on qualitative and quantitative methods and it covers a lot of content from goals and what is a qualitative inquiry right down to specific interview techniques and running focus groups. Again, this module serves as a good introduction and is helpful for those who aren’t quite as familiar with some of these techniques or are looking to brush up
• #46: Next I want to talk about the ethics module and since most of us here today work in a health or health related field, this can be particularly important, especially when you’re dealing with anything which may contain patient information. The module works you through ethical considerations as they pertain to evaluation and how to make sure we’re conducting our evaluations with the highest ethical integrity
• #47: Finally there’s a module on developmental evaluation. Now this module might be a bit more theoretical and not as applied as two of the other modules, but the concepts covered become particularly important when we’re talking about evaluations
• #50: Here’s a profile and along the top we have tabs for different sorts of information you might need
• #51: Finally there’s our project showcase and what this is, is it’s a database of final reports from past evaluation work the Centre has supported with child and youth community agencies across the province of Ontario
• #52: So if we search you get a list of projects you can search through for anything similar and it saves you starting completely from square one