Evaluating Linked Survey and Administrative Data for Policy Research
- 1. Evaluating Linked Survey and Administrative Data for Policy Research Joint Statistical Meetings July 31, 2007 Michael Davern, Ph.D. Assistant Professor, Research Director SHADAC, Health Policy & Management University of Minnesota Supported by a grant from The Robert Wood Johnson Foundation
- 2. Start with conclusions There is great potential for linked survey and administrative data files Survey microdata are in the public domain Strengths and (especially) limitations are well known Administrative data are the standard on programatic issues, but…. Because these data are not in the public domain its imperative that limitations be thoroughly investigated Currently this can only be done by those entrusted with the data. Documentation and research on linked files (e.g., metadata and supporting file documentation) should be put into the public domain even if the microdata are not
- 3. Great potential for linked data Potential for linked data: Improving accuracy of survey data collection of enrollment data (Medicaid, SSI, etc.) Improve survey sample frames (Census MAF) Using linked data to create small area estimates Improve administrative data race/ethnicity information Great benefit to using information in imputation models and editing Improve policy simulations by allowing researchers to better engage errors and appropriately model them
- 4. Survey data have well known limitations Survey data concerns Sample frame coverage error Sampling error and variance estimation Non-response error (item and unit) Measurement error Data processing, imputation/editing Need for better documentation Timeliness and data access How do administrative data files and linked data files compare?
- 5. How do administrative data files and linked files compare? Sample frame and frame coverage The administrative data cover the entire enrolled/filing population For linked data it is important to understand how survey sample frame maps onto the administrative data universe Differing conceptions of “institutionalized” group quarter population Sampling error Not a problem for administrative data because it is a complete list of the enrolled population Linked data files carry with them the sample design of the survey data
- 6. Item non-response and missing data Non-response error (or missing data) Item non-response can be a major issue in administrative data and when linking Important data for research can be missing (e.g., age, address, program codes, or race/ethnicity) Some of these data can be missing systematically E.g, VA study Linking data can also be missing systematically Can be a large source of sample loss when matching survey and administrative data An example of linking the Medicaid data to the Current Population Survey
- 7. Validated Records in MSIS
- 8. Building a common ‘linkable universe’ Institutional Group quarters, dead, People born, non-Medicaid eligible population, Medicaid eligible but unenrolled Medicaid enrollees in linkable CPS and MSIS universe CPS Sample Frame MSIS Medicaid Enrollees
- 9. Measurement error Administrative data are the standard for knowing about programmatic details However they tend to carry a lot of supplementary data that can be more unreliable Administrative data can be collected through many modes, during more than one wave of interviewing, using several instruments Interviewer assisted, self-administered, completely filled out by interviewer enrollees signs, etc. Interviewers have a wide variety of training/skills (e.g. Tax accountants and hospital staff)
- 10. Measurement error (continued) Research is needed into possible mode effects, longitudinal panel conditioning, interviewer effects, and instrumentation effects To do this the administrative data should try to keep track of paradata Important to remember that “respondents” in administrative data files may have different incentives for filling out administrative data versus survey data How do these motivations lead to measurement error/differences? Also if data are not accepted unless filled out, are elements ever “made-up” by data entry folks if unknown? Research on administrative data measurement error is essential
- 11. Data editing and imputation and documentation Data editing and imputation and documentation There is little documentation in the public domain regarding collection, editing and/or imputation procedures of administrative and enrollment data relative to survey data Data editing and imputation activity happens but researchers who use the administrative data files can be caught off guard by it
- 12. Timeliness and Data Access Timeliness of linked data files Linking takes time Most recent linked data for our CPS-Medicaid link is 2004 Data access Due to sensitive nature of both, linking has to be done in a very restricted environment Access to linked data files is, by necessity, very limited Public use linked data files are unlikely US Census Bureau Research Data Centers and/or synthetic data hold promise in this area
- 13. How do administrative data compare to survey data for research purposes? Survey microdata, documentation and research are all in the public domain Survey data are very strong because of the many known (and well documented) problems Similar research into problems with administrative data needs to be done Especially since they are not created for the purpose of research Even though the microdata itself cannot be made public, research and documentation used to produce the file needs to be made public if the data are to be useful for research
- 14. Issues to deal with Essential data stewardship, agreements, confidentiality, privacy concerns need to be worked out Sample loss in linked data files needs to be further studied Need to treat administrative data like survey data and examine measurement error, produce public domain documentation (especially since public use files seem unlikely) Develop standards such as the Data Documentation Initiative for administrative data systems used for research If we do this, the linked administrative and survey data will become even more useful source of data for policy research
- 15. SHADAC contact information www.shadac.org State Health Access Data Assistance Center University of Minnesota 2221 University Avenue, Suite 345 Minneapolis Minnesota 55414 (612) 624-4802