Evaluation of Library STEM Programs: Learning from the BISE Project
0 likes975 views
The document presents findings from evaluating informal science education (ISE) programs, highlighting that most libraries currently do not assess long-term outcomes and rely heavily on qualitative methods. It reveals that only 17% of reports included follow-up data, suggesting a need for libraries to adopt a more comprehensive approach to evaluations, including diverse study designs and data collection methods. The Building Informal Science Education (BISE) project offers resources to help libraries enhance their evaluation practices and engage effectively with the outcomes of ISE projects.
Evaluation of Library STEM Programs: Learning from the BISE Project
- 1. 1% 2% 2% 7% 8% 10% 12% 14% 44% Unknown Home Park, Nature Center or Garden Aquarium or Zoo School, College or University Community Site Online Broadcast Media Museum Evaluation of Library STEM Programs: Learning from the BISE Project Rebecca M. Teasdale • University of Illinois at Urbana-Champaign • rteasd2@illinois.edu 1. Rather than provide a model or standard for libraries to replicate or adapt, this sample presented a rich assortment of study designs and data collection methods to consider. As we set our agenda for evaluation, libraries should consider whether to replicate this variability or to encourage standardized approaches across the field. 2. 83% of the reports in the BISE database did not evaluate mid-term and long-term outcomes. Libraries should ensure that we design studies that examine these effects as well as more immediate outcomes. 3. 70% of the studies in the sample employed qualitative methods. To move beyond libraries’ reliance on survey methods, we should educate librarians about qualitative techniques and a wide range of study designs. We should also engage external evaluators with expertise in a variety of methods and designs. 4. The sample included 35 reports that examined knowledge-related outcomes. Work is underway to identify methods and designs that may be adapted to the library context. 5. Just 3 studies in the sample examined skill- related outcomes. Libraries may be poised to provide leadership in this area due to our large experience with technology instruction and our emerging work in maker programming. l Used the Building Informal Science Education (BISE) database as a window into evaluation of ISE programs. l NSF funded the BISE project to gather and code 520 evaluation reports that were voluntarily posted to informalscience.org. l Several BISE products are freely available at VisitorStudies.org/BISE and can help librarians learn about ISE evaluation: Friedman, A. (Ed.) (2008). Framework for evaluating impacts of informal science education projects. Retrieved from http://informalscience.org/documents/ Eval_Framework.pdf VisitorStudies.org/BISE www.informalscience.org Background Resources MethodsQuestions Amy L. Grack Nelson • Science Museum of Minnesota • agnelson@smm.org Step 1: Drew sample of evaluations from the BISE database that examined mid-term or long-term outcomes. Definition: Effects that manifest days, weeks, months, or years after an informal learning experience. Requires collecting data some time after the participant engages in the learning experience. 1. What types of participant outcomes do evaluators of informal science education (ISE) projects study? 2. What study designs and methods do evaluators employ to examine those outcomes? 3. As public libraries shape our agenda for STEM program evaluation, what can we learn from outcome evaluation of other ISE projects? Just 17% of reports in the BISE database (n=520) included follow-up data collection No follow up 83% Follow up 17% Initial Findings 2%3% 8% 42% 67%68% Document review TestFocus groupObservationSurveyInterview A wide variety of study designs and data collection methods were found in the sample (n=60). Interviews and surveys were the data collection methods used most frequently. 52%53% InterviewSurvey In the sample (n=60), follow up data were collected through surveys and interviews. 64% 44% 31% 20% 4% Awareness, knowledge or understanding Behavior Attitude Engagement or interest Skills All five Friedman (2008) ISE impact categories were studied in the sample (n=60). Knowledge gain was examined most frequently. Implications for Libraries p File of the 520 reports: Explore ISE evaluation designs, methods and findings. p NVivo database of coded reports: Search for examples that match a specific project or need. p Instructions and worksheets: Learn to use NVivo. Think through ways to use BISE resources to plan an evaluation or learn more about ISE studies. 70% of the reports in the sample (n=60) included qualitative methods in the study. Qualitative methods 70% No qualitative methods 30% Most of the projects evaluated in the sample (n=60) that took place in a museum. Step 2: Narrowed to studied that investigated outcomes related to public audiences. Definition: Members of the public who engage in informal learning experiences such as individuals who visit a museum or watch an educational TV show. Does not include professional audiences such as museum employees, teachers, librarians, or scientists.