Survey Design Series
- 1. Developing Survey Goals Smart Start Funded Programs use their Contract Activity Design (CAD). Non Smart Start funded programs use their agency goals aligned with PFC goals PFC Goals PBIS (Outcomes) PSC (Activity/strategy) Goal 1 Health Goal 4 System Support Goal 3 Early Care and Education Goal 2 Family Support Early Intervention Health Care Compliance Child Care Placements Staff Education Staff Benefits Stability Parenting Skills Family Literacy Early Intervention/Spec. Ed Family Support Pre-K Classes Parent Education Literacy programs CCR&R, More at Four, Subsidy, Quality Enhancement/Maintenance Training, Prof. Development Program Eval/Coordination, System Integration, Community Outreach, Family Resource Center Predict needs of client, community or system Measure satisfaction with programs, services Determine change in community or client behaviors, perspectives Evaluate program effectiveness Assess program impact on client, community, system Gauge knowledge gained and implemented These goals overlap; they are not mutually exclusive. Many can be examined within one survey
- 2. Closed Ended (Quantitative) Open Ended (Qualitative) Responses are researchers ‘acceptable’ choices Makes responses socially acceptable Responses are mutually exclusive Narrative form replaces vague responses Range of responses exceeds appropriate question length Measure preferences or behaviors Measure knowledge (pre-or post) Questions and answers are consistent and understandable Questions are administered to respondents in the same manner each time Respondents have access to information needed to answer the question Respondents are willing to provide answers Five Challenges to Writing Good Questions Defining objectives; specifying answers to meet those objectives Ensuring respondents have shared, common understanding of question meaning Respondents are asked questions that they can answer Respondents can answer in terms required by the question Questions are asked in a way respondents are willing to answer Determine Survey Goals : Smart Start Contract Activity Design (CAD) or agency/program goals Determine Survey Scope : Inquiry, Evaluation or Predictive Survey Question Development Question and responses are standardized
- 3. Children 0-5 years old and not in Kindergarten Children 5 yrs + and in Kindergarten Cumberland County (all persons) Households with children Households without children Persons interested in the well-being of children, families, schools and community Children and Health Children and Early Care & Education Children and Families Children and System Support Subsets of Children Subsets of Community (With or without children) Legislator/Policy Makers School Personnel (K-12) Direct Service Providers Community Leaders/Stakeholders Determining Survey Populations
- 4. Design Analysis Evaluation Survey Question Bank Centralized database of survey questions aligned with Performance Based Incentive Standards (PBIS) and Purpose Service Codes (PSC) Aid in survey design Questions are evaluated, pre-tested Streamline data collection Outputs track service area progress Smart Start, local agencies evaluate similar areas Pre-coded responses Continuity in question content Continuity in question format Organization by topic area 4-stage evaluation process 1-year trial PFC goal, PBIS, PSC grouping By time and individual question Linked by age cohort, service areas Construct Validity Correlation of several questions asked in a similar way Predictive Validity Measures that predict answers to other related questions. Discriminant Validity Extent of differentiation in expected responses
- 5. Customers/clients - Are Smart Start grantees Are non-Smart Start funded Serve children/families in K-12. Customer/clients identify their programs’ primary focus as - Health Early care and education Family support System support Surveys are designed by R&D Surveys are analyzed by R&D Questions are created annually supporting - Smart Start Purpose Service Code (PSC) Smart Start Performance Based Incentive System (PBIS) Customers/clients report – Selecting questions from the survey database. Creating their own surveys. Question bank is current in outputs and research. Customers/clients report - Increased understanding of survey design, evaluation Strong satisfaction or satisfaction with survey design and evaluation technical assistance. Learning 1-new tool/skill Implementing the new/tool in their programs Referring this resource to others Strengthen program and community data collection through integrative survey development processes Objective 2: A centralized database of questions and templates for survey design. Objective 3: Customers/clients use surveys to collect program data. Objective 1: Data is collected from a variety of community organizations. PFC Assumptions Aligning Smart Start goals and outcomes with survey questions supports targeted evaluations. Pre-determined questions assist providers in coordinating survey goals with desired outcomes. Providing technical assistance in survey design and analysis increases reliable data collection. Data collection from like-mission agencies reinforces tracking systems with similar outcomes. Demystifying survey design and analysis increases the use of surveys in program evaluation.