Return on Investment and Performance Measurement of Workforce Development Programs
Download as PPTX, PDF1 like581 views
The document summarizes research on measuring the return on investment and performance of workforce development programs. It discusses methods for establishing causality between programs and outcomes. It then presents results from studies on the net impacts and returns on investment of various workforce programs in Washington state and other states. These include impacts on employment, earnings, skill attainment rates, and returns for participants, taxpayers, and society. The document concludes by noting new requirements in WIOA for indicators to measure employer satisfaction with workforce programs.
Return on Investment and Performance Measurement of Workforce Development Programs
- 1. Return on Investment and Performance Measurement of Workforce Development Programs Kevin Hollenbeck Vice President, Senior Economist W.E. Upjohn Institute for Employment Research hollenbeck@upjohn.org February 25, 2015 Presented at OERC Conference: Workforce Outcomes: Informing Policy & Practice through Education & Labor Market Data, Columbus, Ohio
- 2. Outline I. Causality – How do we know what we know? II. Net Impact of Washington’s Workforce Development System III. Return on Investment IV. Other States’ Results V. Employer Satisfaction Performance Standard VI. Conclusions
- 3. Causality Washington Legislation • The Workforce Board is required by Washington State RCW 28C.18.060(10) to: Every two years administer scientifically based outcome evaluations of the state training system, including, but not limited to, surveys of program participants, surveys of employers of program participants, and matches with employment security department payroll and wage files. Every five years administer scientifically based net- impact and cost-benefit evaluations of the state training system. (Emphasis added.)
- 4. Causality (Scientifically based) Physical Science: If A B Social Science/Education: If A usually B How do you verify/falsify? 1) Large sample 2) Counterfactual (Absence of A) – Do you usually get B anyway?
- 5. Causality State of the Art: 1) Randomized Control Trials (RCTs) 2) Quasi-experimental (Statistical Matching) requires an assumption about nonobservables 3) Econometric Models
- 6. Washington Net Impact Results Treatment Comparison Exited from a program in fiscal years 2005/06 or 2007/08 Encountered Job Service in fiscal years 2005/06 or 2007/08
- 7. Table 1 Short-Terma Net Impacts of Washington’s Education and Training System, by Program Program Net Employment Impact (in percentage points) Net Hourly Wage Impacts (‘05 $) Net Hours Employed per Quarter Impacts Net Quarterly Earnings Impacts (‘05 $) Net Impact on the Rate of Receiving UI (in percentage points) WIAAdults 12.8%*** $1.50*** 73.6*** $1,559*** –1.7%*** WIA Dislocated Workers 10.1%*** –$0.96*** 42.4*** $ 44 –3.1% *** WIA Youth 8.0%*** –$0.59** 46.5*** $ 359*** 0.3% *** Comm. College Job Prep 6.6%*** $2.75*** 59.8*** $1,856*** –4.7%*** Comm. College Worker Retraining 8.8%*** –$0.15*** 26.6*** $ 367*** –0.1%*** Comm. College ABE –2.1%*** $0.22*** 15.4*** $ 210*** –0.5%*** IBEST 3.9%*** $1.12*** 35.3*** $ 526** –1.2%*** Private Career Schools –2.7%*** $0.94*** 20.8*** $ 558*** –3.2%*** Apprenticeships 7.8%*** $7.17*** 46.9*** $4,216*** 0.7%*** High School Career Technical Ed. 6.0%*** –$0.21*** 19.2*** $ 173*** –0.2%*** Vocational Rehab. 12.8%*** $0.05*** 1.5*** $ 286** –1.8%*** Div. of Services to Blind na na na na na aDefined as third full quarter after quarter of exit. na -- not available for this draft. ***,**,*Statistically significant at the 0.01, 0.05, 0.10 levels NOTE: Estimates are preferred specifications as “boxed” in Hollenbeck and Huang (2006) tables. They are regression-adjusted estimates of impact levels for 2007/2008 cohorts of WIA Dislocated Workers, WIA Youth, Community College Worker Retraining, and High School Career and Technical Education. They are regression-adjusted difference-in- difference net impact estimates for all other programs. Washington Net Impact Results
- 8. Table 2 Long-Terma Net Impacts of Washington’s Education and Training System, by Program Program Net Employment Impact (in percentage points) Net Hourly Wage Impacts (‘05 $) Net Hours Employed per Quarter Impacts Net Quarterly Earnings Impacts (‘05 $) Net Impact on the Rate of Receiving UI (in percentage points) WIAAdults 10.8%*** $1.46*** 43.6*** $ 952*** –1.9%*** WIA Dislocated Workers 4.7%*** $1.28*** 28.4*** $ 756*** –3.2%*** WIA Youth 4.3%*** $0.10*** 30.8*** $ 429*** 2.2%*** Comm. College Job Prep 10.1%*** $2.91*** 59.5*** $1,976*** –5.7%*** Comm. College Worker Retraining 7.5%*** $0.91*** 23.5*** $ 627*** –0.9%*** Comm. College ABE –3.9%*** –$0.06*** 15.1*** $ 189*** –5.6%*** Private Career Schools 3.4%*** $0.56*** 27.1*** $ 470*** –4.4%*** Apprenticeships 9.8%*** $7.97*** 26.4*** $4,019*** 5.6%*** High School Career Technical Ed. 8.4%*** $0.32*** 32.2*** $ 450*** 2.1%*** Vocational Rehab. 12.4%*** –$0.18*** 47.4*** $ 305** 0.6%*** Div. of Services to Blind na na na na na aDefined as average over quarters 9 - 12 after quarter of exit. na -- not available for this draft. ***,**,*Statistically significant at the 0.01, 0.05, 0.10 levels Note: Estimates are preferred specifications as “boxed” in Hollenbeck and Huang (2006) tables. They are regression-adjusted estimates of impact levels for 2005/2006 cohort of WIA Dislocated Workers, WIA Youth, Community College Worker Retraining, and High School Career and Technical Education. They are regression-adjusted difference-in- difference net impact estimates for all other programs. Washington Net Impact Results
- 9. Washington Net Impact Results What Does State Do with Results?
- 10. Return on Investment (Theory) Benefits Participants Taxpayers Society Higher productivity 0 + + Higher earnings + 0 + Fringe benefits + 0 + Less unemployment/turnover + + + Lower income maintenance transfers - + 0 Higher taxes - + 0 Costs Tuition, fees -/0 - - Forgone wages - 0 - Total (Net Benefits) + ?? +/??
- 11. Return on Investment (Actual Framework) Benefits Participants Taxpayers Society Higher productivity 0 + + Higher earnings + 0 + Fringe benefits + 0 + Less unemployment/turnover + + + Lower income maintenance transfers - + 0 Higher taxes - + 0 Costs Tuition, fees -/0 - - Forgone wages - 0 - Total (Net Benefits) + ?? +/?? Note: Entries in the table that are highlighted are omitted from empirical estimation.
- 12. Return on Investment – Secondary CTE Table 4 Participant and Taxpayer Benefits and Costs per Completer in Secondary CTE Programs Benefit/Cost First 2.5 years Lifetime (until 65) Participants Taxpayers Participants Taxpayers Benefit Earnings Fringe Benefits Taxes 2,753 551 −475 0 0 475 34,603 6,920 −5,969 0 0 5,969 Transfers UI TANF FS Medicaid 16 −91 −73 −27 −16 91 73 27 168 −442 −380 −169 −168 442 380 169 Costs Forgone earnings Program costs −28 0 0 704 −28 0 0 704 NOTE: ’00 $. Discount rate of 3% real. SOURCE: Hollenbeck and Huang (2006).
- 13. Return on Investment – Postsecondary CTE Table 3 Participant and Taxpayer Benefits and Costs per Participant in Postsecondary CTE Programs Benefit/Cost First 2.5 years Lifetime (until 65) Participants Taxpayers Participants Taxpayers Benefit Earnings Fringe Benefits Taxes 10,386 2,077 −1,792 0 0 1,792 79,239 15,848 −13,669 0 0 13,669 Transfers UI TANF FS Medicaid −2,137 351 107 45 2,137 −351 −107 −45 −2,629 933 331 161 2,629 −933 −331 −161 Costs Forgone earnings Program costs 2,100 3,519 0 6,877 2,100 3,519 0 6,877 NOTE: ’00 $. Discount rate of 3% real. SOURCE: Hollenbeck and Huang (2006).
- 14. Return on Investment Table 5 Benefits, Costs, and Annual Rates of Return for Postsecondary and Secondary CTE Programs over the First 2.5 Years and Lifetime for the Average Participant Benefit/Cost Secondary CTE Postsecondary CTE First 2.5 years Lifetime (age 65) First 2.5 years Lifetime (age 65) Participants Benefits Costs irr 2,654 -28 na 34,731 -28 na 9,037 5,619 20.93% 80,204 5,619 8.81% Taxpayers Benefits Costs irr 650 704 -3.14% 6,792 704 4.70% 3,431 6,877 -24.28% 14,873 6,877 2.48% Society Benefits Costs irr 3,304 676 88.64% 41,523 676 25.35% 12,468 12,496 -0.09% 95,077 12,496 6.65% NOTE: Table entries are for average participant. Benefits include earnings, fringe benefits, and income-related transfer payments. Costs include tuition and fees (if any), foregone earnings, and public program costs per participant. $ figures are in real 2000$. na means that irr could not be calculated because of 0 or negative costs. Discount rate is 3% real.
- 15. Return on Investment • Secondary and postsecondary CTE have huge payoffs for participants, especially in short-run • State/taxpayers eventually get positive ROI, but payback takes slightly more than 2.5 years for secondary and considerably longer for postsecondary • “Social” ROI quite positive for secondary and for postsecondary
- 16. Results from Other States Table 5 A Comparison of Findings Across States Outcome WIA-Adult WIA-Dislocated Workers WIA-Youth TAA Postscondary Education Employment in 3rd quarter (%) Indiana 14.8*** 17.0*** 3.4 3.2 17.9*** Washington 9.7*** 8.7*** 4.2** na 10.3*** Virginiaa, c 3.4*** −3.9** −5.9*** 2.8*** Quarterly earnings in 3rd quarterb ($) Indiana 549** 410** 24 −122 1,490*** Washington 711*** 784*** 66 na 1,275*** Virginiaa, c 146*** 62 −154*** 1,539*** NOTES: *, **, *** impact estimate is significant at 0.10, 0.05, and 0.01 level. aWIA-Adults and WIA-Dislocated Workers were combined in the study. bIndiana results in 2008$; Washington and Virginia in 2005/2006$. cVirginia results are for the 4th quarter after exit.
- 17. Results from Other States Table 6 Net Impacts, Disaggregated by Training Status Outcome WIA-Adults WIA-Dislocated Workers All Training No Training All Training No Training Employment, 3rd Qtr (%) 14.8*** 19.2*** 9.5*** 17.0*** 15.4*** 18.3*** Employment, 7th Qtr (%) 13.7*** 18.2*** 8.2*** 16.5*** 15.9*** 17.0*** Earnings, 3rd Qtr ($) 549*** 751*** 339*** 410*** 482*** 354*** Earnings, 7th Qtr ($) 463*** 692*** 221*** 310** 394*** 245*** UI benefits, 3rd Qtr ($) −15 −17 −12 −53** −70** −39** UI benefits, 7th Qtr ($) 10 1 21 3 −20 21 NOTES: *, **, *** impact estimate is significant at 0.10, 0.05, and 0.01 level.
- 18. Results from Other States Table 7 Quarterly ROIs Using Lifetime Earnings Flow Estimates WIA-Adult WIA- Dislocated Workers WIA-Youth TAA Postsecondary Individual program participant (%) 16.32 2.64 13.27 −0.93 29.87 Government (%) −0.04 1.50 −1.73 5.01 1.82 Society (takes into consideration individual and government) (%) 7.60 2.13 0.22 −0.40 9.66
- 19. Results from Other States Table 2. Net Impact Indicators for Virginia’s Workforce Programs, FY2005 Program Short-term employment/ in school rate Long-term employment/ in school rate Short-term earnings level Long-term earnings level Credential completion rate DOE and VCCS programs AEL (DOE) Postsecondary CTE (VCCS) −9.58*** 0.49 −9.07*** 2.81*** 289*** 1,213*** -21 1,539*** 65.48*** 22.68*** DRS and DBVI programs DRS DBVI 17.63*** 17.37*** 16.17*** 25.00*** 429*** 1,948*** 241*** 1,318** 8.79*** 6.78*** DSS programs FSET TANF/VIEW −9.29*** −2.22*** −9.55*** −2.30*** −404*** 414*** −529*** 175* −0.41*** 0.46*** VEC and Senior Advisor programs TAA (VEC) WIAAdultsa (Senior Advisor) WIA Youth (Senior Advisor) −6.15*** 4.75*** −2.91* −5.88*** 3.39*** −3.88** −210*** 442*** 480*** −154*** 146** 62 65.03*** 53.96*** 76.12*** W/P (VEC) post–pre impactsb −0.36 0.09 −250*** 38* na aIncludes WIA Dislocated Workers. bW/P program impacts were computed using a post-pre methodology as described in text. na means not available because credential completion is not meaningful in a post-pre methodology. Significance levels (one-tailed test): * 10%; ** 5%; *** 1%.
- 20. Results from Other States Other Studies: • Heinrich, Mueser, Troske (2008) – 12 States WIA Adult and Dislocated Worker Titles • Hollenbeck, Schroeder, King, Huang (2005) – 7 states WIA Adult and Dislocated Worker Titles
- 21. Employer Satisfaction Performance Standard WIOA (Title I, Chapter 4, Section 116 (b)(A)(iv)) (iv) INDICATOR FOR SERVICES TO EMPLOYERS.—Prior to the commencement of the second full program year after the date of enactment of this Act, for purposes of clauses (i)(VI), or clause (ii)(III) with respect to clause (i)(IV), the Secretary of Labor and the Secretary of Education, after consultation with the representatives described in paragraph (4)(B), shall jointly develop and establish, for purposes of this subparagraph,1 or more primary indicators of performance that indicate the effectiveness of the core programs in serving employers.
- 22. Employer Satisfaction Performance Standard Table 1. Gross Impact Indicators for Virginia’s Workforce Programs, FY2005 Program Short-term employment/ in school rate Long-term employment/ in school rate Short-term earnings level Long-term earnings level Credential completion rate Percent of repeat employer customers DOE and VCCS programs AEL (DOE) 59.99 59.38 3,572 3,824 66.19 50.46 Postsecondary CTE (VCCS) 72.76 73.01 5,426 6,064 25.48 45.15 DRS and DBVI programs DRS 54.85 53.98 2,822 3,052 20.19 46.07 DBVI 38.13 40.13 4,176 4,083 19.73 6.98 DSS programs FSET 54.93 53.16 2,717 2,961 0.20 68.59 TANF/VIEW 63.34 61.85 3,211 3,410 1.11 58.04 VEC and Senior Advisor programs TAA (VEC) 67.93 68.46 4,366 4,965 65.02 45.40 W/P (VEC) 71.15 70.14 4,257 4,729 0.65 84.58 WIAAdultsa (Senior Advisor) 75.98 74.12 4,439 4,733 54.91 51.86 WIA Youth (Senior Advisor) 71.61 67.04 1,637 1,761 81.39 49.20 aIncludes WIA Dislocated Workers. NOTE: The workforce programs vary considerably in size. Table C.2 shows the number of records of exiters that were supplied to us by program.
- 23. Conclusions Have we identified causality? To RCT purists, the answer is no. But suppose we use a “probable cause” standard, then consistency across study results may meet that standard: • JTPA/WIA works for disadvantaged adults • Postsecondary CTE works • Voc. Rehab. works • Apprenticeship works
- 24. Conclusions (Continued) • Adult basic education “struggles” (labor market outcomes may not be most important) Conflicting Results • Dislocated workers • WIA Youth
- 25. Conclusions (Continued) My advice: • Never make a decision based on a single study (too many parameters, other sources of uncertainty) • However, (administrative) data can: i) Be used for program improvement by looking at trends and subgroups ii) Be used for accountability and performance measurement because they can be used, at minimal expense, to examine trends and subgroups.
- 26. Thank you!