Chapter IV. DATA COLLECTION INSTRUMENTS
- 1. UNIVERSITY OF THE ARMED FORCES SOCIO-EDUCATIONAL PROJECT DESIGNING EDUCATION PROJECTS Part IV. DATA COLLECTION INSTRUMENTS DAYANA CRISTINA TAMAYO
- 2. DATA COLLECTION INSTRUMENTS INTRODUCTION Evaluators can use different data collection instruments according some considerations in order to select the most appropriate These are some of the considerations to take into account: Type of evaluation (need assessment, process, outcome) Level of evaluation (KASA changes, practices, SEE conditions) The audience (children, adults, casual or organized groups) The amount of resources available (time, money)
- 3. DATA COLLECTION INSTRUMENTS Matching Data Collection Instruments to What is Being Assessed Some specific data collection instruments can evaluate or document the level of skills, behavior, attitude development or knowledge gain. For example: Interview: evaluates knowledge, attitude, and maybe behavior Observation: assesses skills and behavior Questionnaire: evaluates knowledge, skills, attitude, and maybe behavior
- 4. DATA COLLECTION INSTRUMENTS Matching Data Collection Instruments to What is Being Assessed Literature Review: assesses knowledge, skills, attitude, and behavior Test: evaluates knowledge and skills Concept maps: evaluates knowledge, and maybe attitude
- 5. DATA COLLECTION INSTRUMENTS Matching Data Collection Instruments to What is Being Assessed Focus group: evaluates attitude, and maybe knowledge Document or product review: evaluates knowledge, skills, and maybe attitude Case study: assesses knowledge, skills, attitude, and behavior
- 6. When selecting the data collection instruments for project evaluation, it is necessary to validate them before to use, taking steps to ensure instruments are valid and reliable through two considerations: DATA COLLECTION INSTRUMENTS Validity and Reliability of Evaluation Instruments CONSIDERATIONS Validity Reliability
- 7. Validity: it is the extent to which it measures what it pretends to measure Reliability: it is the extent to which it yields responses when is administered DATA COLLECTION INSTRUMENTS Validity and Reliability of Evaluation Instruments
- 8. DATA COLLECTION INSTRUMENTS Validity and Reliability of Evaluation Instruments Field or Pilot testing Evaluators • can measure validity and reliability of evaluation instruments • test a representative sample of project participants with instruments selected to gather information
- 9. DATA COLLECTION INSTRUMENTS Validity and Reliability of Evaluation Instruments Field or Pilot Testing The field test should answer to: Necessary information to answer evaluation questions The way of collecting information Consistency and accuracy in information obtained Although there are other techniques to validate instruments using statistical methods, but they depend of: expert evaluation team data collection methods project’s longevity and practicality
- 10. DATA COLLECTION INSTRUMENTS MIXED METHODS The using of a combination of data gathering like quantitative and qualitative in evaluation. Using both type of instruments can provide more accuracy and validity of the information and results from evaluation, because each instrument has individual strengths and weaknesses, and using together, the weaknesses are minimized. QUANTITATIVE QUALITATIVE MIXED METHODS
- 11. DATA COLLECTION INSTRUMENTS MIXED METHODS QUANTITATIVE METHODS Count events and number of participants Determine cost/participants Perform statistical analyses Generalize findings beyond actual respondents
- 12. QUALITATIVE METHODS Record participants’ own words about experiences (perceptions, descriptions) Create narratives in-depth view and understanding of the evaluation context Participate few participants and cannot generalize results DATA COLLECTION INSTRUMENTS MIXED METHODS
- 13. DATA COLLECTION INSTRUMENTS Types of Data Collection Instruments Many of the data collection instruments are oriented to a specific purpose and project, having advantages and challenges of using them. There are 9 data collection instruments: understand someone’s experiences provides full range of information be hard to analyze and compare INTERVIEWS explore a topic in-depth quickly obtain common impressions hard to analyse responses FOCUS GROUP obtain lot of information from people complete anonymously doesn’t yield full story QUESTIONNAIRES
- 14. DATA COLLECTION INSTRUMENTS Types of Data Collection Instruments gather accurate information adapt to events as they occur be difficult to interpret behaviors OBSERVATION gather information on the audience provides evidence about the problem may not address specific questions LITERATURE REVIEW determine audience’s current state of knowledge results are easily quantified limited availability TESTS
- 15. gather information of someone’s understanding be a better tool for visual learners be challenging and time consuming to score CONCEPT MAPS gather with information already exists yields historical information takes much time DOCUMENT OR PRODUCT REVIEW understand client’s experiences in a project powerful means to portray project to outsiders Information cannot be generalized CASES STUDIES OR PEER REVIEW DATA COLLECTION INSTRUMENTS Types of Data Collection Instruments
- 16. DATA COLLECTION INSTRUMENTS Selecting the Right Data Collection Instrument According the type of project, activities and the information needed the project coordinator can select the appropriate evaluation data collection instrument. By the audience: Adults (interview, focus group, survey, observation, test, case study) Cultural groups (focus group, observation) Teens (interview, observation, test, concept map, case study) Children (observation, test, concept maps) By the project/activity: Talk/lecture (test, survey, concept map) Workshop (survey, test, interview, focus group, concept map) Training (interview, focus group, observation, survey, test) Exhibit (interview, focus group, survey, observation, concept map) Media (interview, focus group, survey) Interactive media (interview, focus group, observation, concept map)