Graphical interactive tool for evidence based public health program evaluation
- 1. A Graphical Interactive Tool for Evidenced Based Public Health Program Evaluation by Programmatic Personnel with Minimal Computer Skills Leisa J. Stanley, MS Healthy Start Coalition of Hillsborough County, Inc. David Darr, MD, MPH Lawton & Rhea Chiles Center, College of Public Health, University of South Florida 12th Annual MCH Epidemiology Conference Atlanta, Georgia December 6-8, 2006
- 2. Project Background Healthy Start Coalitions – HSC of Hillsborough County – Established by legislative act, Chapter 383, FS., 1991 – Responsibilities for MCH System Chiles Center for Healthy Mothers & Babies – Part of Department of Community & Family Health, College of Public Health, USF – The Chiles Center brings together experts in public health, health care, program design and evaluation, and maternal and child health from Florida universities and state, national and international programs. Healthy Start Program – Countywide Program – 16,000 births in 2005 – In 2005-2006, served 6,000 pregnant women and 7,000 infants – Provides home visitation and care coordination services – 12 state and 11 local Performance Measures – 6 state Outcome Measures
- 3. Problem Statement Question: Can data be prepared, so it is easily and reliably used by non-programmers for programmatic evaluation with software found on virtually all computers? Issues: – Data Access: vital statistics, service data – Data Timeliness – Data Volume – Staff Training System-Wide Issues – Increasing number of pregnant women without insurance – Rising Infant Mortality Rate – Racial Disparity
- 4. Purpose Creation of database with historical data: – 1992 – forward – Vital Statistics Data – Healthy Start Services Data Addition of provisional vital statistics data – Timeliness Two Pathways – Research Database – Program Evaluation Database
- 5. Creation of a Database and Interactive Tool
- 6. Data Files • Florida Department of Health Office of Vital Statistics • Natality Record • Fetal Death Records • Infant Death Records • Provisional Vital Statistics Records • Florida Department of Health Office of Planning & Evaluation • Healthy Start Prenatal Screening Data • Healthy Start Infant Screening Data • Healthy Start Services Data • Hillsborough County Health Department • Local Healthy Start Prenatal Risk Screening Data • Healthy Start services data with supplemental local service codes
- 8. Data Manipulations Converted all month/day/year variables into true SAS dates Caculated needed variables – Kotelchuck Index – Interpregnancy Intervals Created as many categorical variables as possible, with as many variations as possible to reduce the potential need for future reprogramming – Race/Ethnicity – Age of Mother – Birth Weight – Gestation
- 9. Data Manipulations Created a variable to distinguish between a delivery (maternal data) and a birth (infant data) Defined and created variables for service areas or targeted populations – Central Hillsborough Healthy Start – Intensive Teen Unit – Defined geographic regions Created value-added variables (Yes/No) – Low Birth Weight – Preterm Birth – Infant Death – Early Entry into Prenatal Care
- 10. Data Manipulations Created Pivot Charts in Excel – Data dictionary is available – Programming already done – Limitations: 256 variables; 65,563 records Created macros in SAS to calculate from data and import back into Excel – Calculated Relative Risks Agree on reference groups – Calculated Attributable Risk – Calculated Population Attributable Risk
- 11. ADEQUATE ADEQUATE PLUS INADEQUATE INTERMEDIATE 2001 2002 2003 0 50 100 150 200 250 300 350 400 450 500 2001 2002 2003 CHHS_CENSUS_YN YES Count of IDNUMBER KOTELCHUCK_INDEX YEAR
- 12. CHHS_CENSUS_YN YES Count of IDNUMBER YEAR KOTELCHUCK_INDEX 2001 2002 2003 Grand Total ADEQUATE 168 150 181 499 ADEQUATE PLUS 467 465 450 1382 INADEQUATE 106 81 79 266 INTERMEDIATE 9 9 6 24 Grand Total 750 705 716 2171 Sample of a Pivot Table
- 14. Uses by Programmatic Personnel Used by Federal Healthy Start Project Allows complex questions to be answered easily – No programming experience is required – Minimal training is required Questions – Racial Disparity Gap – Repeat Pregnancies for Teens – Entry into Prenatal Care Prepare reports for funders and key stakeholders
- 15. Data Available versus Data Needed Available – 1998-2003 de- identified linked vital statistics data – 1992-2004 de- identified unlinked vital statistics data – 2001-2003 service data (incomplete) Needed – Identified vital statistics data to link – 1992-present complete services data – Local service data codes – Provisional vital statistics data
- 16. Data Procurement Apply for Institutional Review Board Approval – University of South Florida – Florida Department of Health – Requires execution of proprietary data usage agreement(s) with each entity owning the data desired for linkages – Requesting identified data and development of a research database When we did not need to apply? – For programmatic evaluation
- 17. Provisional Vital Statistics Data Specific program evaluation questions: – Issues of program delivery – Issues of access to care – Emerging areas in county Do not use for: – Outcome evaluation – Be cautious with low numbers and impact on rates: Wide month to month variations are possible and small numbers preclude use in outcomes analyses
- 18. Lessons Learned Initiate process as early as possible – Anticipate delays, especially in use of provisional vital statistics data – If identifiers are to be retained, yearly data usage agreements may be necessary Understand limitations of provisional vital statistics data
- 19. Lessons Learned Data Security – Assure original data owner that only de-identified data will be used by programmatic personnel for their evaluations – Assure original data owner that there will be no follow-back to any client – Assure original data owner that all results will be reported in aggregate only – Be very mindful of HIPAA rules, especially when reporting at zip code or census tract levels for rare conditions or events Do not allow original data owner to have veto power over any reporting of data Know and understand the data you are given
- 20. Contact Information Leisa J. Stanley, MS lstanley@hstart.org David Darr, MD, MPH ddarr@health.usf.edu