Public Safety Hiring Tutorial
- 1. Public Safety Hiring: A Tutorial on Improving Your Selection Process From the RFP to the First Day of the Academy John M. Ford, Ph.D. Chris W. Hornick, Ph.D. Kathryn A. Fox, M.A. IPAC 2014
- 2. Important Considerations Before You Start Developing (or Evaluating) Your Process 2
- 3. Factors That Should Drive Public Safety Hiring Decisions Hiring Decisions Professional Standards Legal Guidelines Scientific Literature Practical Experience Best Interests of Organization Best Interests of Community 3
- 4. Factors That Should Not (But Often Do) Drive Public Safety Hiring Decisions 4 Hiring Decisions Political Pressure Past Precedence Archaic or Misguided Rules Self Interest Reactive Emotional Responses Rumors Fear
- 5. Define Your Selection Process Goals 5 Hiring Process Goals Quality Candidates Legal Defensibility Candidate / Public Acceptance Cost & Required Resources Diversity
- 6. 6 Evaluate the Scope of Your Job Analysis • Evaluate the extent to which your job analysis encompasses the complete job domain. – Ensure that your job analysis captures the full range of KSAOs relevant to the job. – Identify the broad range of non-task elements that employees need to be successful. • Ensure that technical and task-related KSAOs are grouped similarly to how non-technical and non-task KSAOs are grouped. – Over emphasis on “knowledge of” and task-related abilities will drown out other critical elements of the job.
- 7. Example – Importance Ratings of a Firefighter Job Analysis with a Consortium of Fire Departments Very Important Essential for successful performance of the job Critically Important Failure to perform results in extreme negative consequences 3.8 4.3 4.3 4.3 4.4 3.8 4.3 4.3 3 3.5 4 4.5 Basic Educational Skills Emotional Outlook Interpersonal Skills Practical Skills Physical Skills Mechanical Aptitude Firefighting Skills EMS Skills
- 8. Example – Frequency Ratings of a Firefighter Job Analysis with a Consortium of Fire Departments 3.6 4.2 4.3 4.2 4 3.4 3.6 4.2 3 3.5 4 4.5 Basic Educational Skills Emotional Outlook Interpersonal Skills Practical Skills Physical Skills Mechanical Aptitude Firefighting Skills EMS Skills Fairly Often Approximately once per shift Very Often More often than once per shift
- 9. Example – Necessity Ratings of a Firefighter Job Analysis with a Consortium of Fire Departments 3.8 4.2 4.1 4 4.3 3.1 2.7 3.3 2.5 3 3.5 4 4.5 Basic Educational Skills Emotional Outlook Interpersonal Skills Practical Skills Physical Skills Mechanical Aptitude Firefighting Skills EMS Skills Required at a minimum performance level at time of hire Required at a full performance level at time of hire
- 10. 10 Example – Police Officer Job Analysis Linkage Results 55% 85% 13% 70% 23% 0% 25% 50% 75% 100% Basic Educational Skills Emotional Outlook Interpersonal Skills Practical Skills Physical Skills % of Relevant Job Duties Linked to KSAOs
- 11. RFPs / Evaluating Proposals 11
- 12. Take Control of Your Proposal Evaluation Process • The way many proposal evaluation processes currently function is not conducive to making informed decisions. – Don’t let test providers decide what information you should consider. • Evaluating tests requires professional judgment. – You must ask the right questions and evaluate the evidence.
- 13. Write Clear and Specific RFPs • Define and prioritize your organization’s goals with regard to testing. • Clearly define the project (or clearly specify when you want guidance). • Allow consultants to demonstrate their strengths and the value they can provide. • Define your evaluation criteria based on your values. • Determine appropriate weights for evaluating or scoring tests. • Give yourself adequate time to review tests. • Don’t be afraid to ask follow-up questions. 13
- 14. Proposal Evaluation • Select proposals that utilize assessment tools that are consistent with your job analysis results. • Evaluate the strength of the validation evidence. • Pay attention to samples. • Don’t compare corrected validity coefficients to uncorrected validity coefficients. • Evaluate adverse impact in a manner that is consistent with how you will use the test. • Recognize that adverse impact in applicant samples will be larger than in incumbent samples. • Has the test been involved in litigation? • Don’t just evaluate the tests, evaluate the consulting company / test provider. 14
- 15. Remember Hidden Costs • Poor performance • Increased training/remedial training/retraining • Lawsuits • Turnover • Grievances • Disciplinary problems • Accidents
- 16. Hiring Guidelines & Policies 16
- 17. Some Agencies Have Policies that Function as a Bureaucratic Straightjacket • Many agencies have policies or agreements that hinder their ability to develop an effective hiring process. – Mandatory Cut Scores – Mandated Selection Tools – Mandated Preference Points – Mandated Weighting of Components – Banding Requirements – Mandated Minimum Requirements 17
- 18. Strategies for Changing Policies & Agreements • Build long-term relationships. • Establish your credibility. • Demonstrate fairness and lack of bias. • Show that you have their best interests at heart. • Learn the organization (organizational structure, key decision makers and potential advocates, available resources, culture and values, lingo, etc.). • Identify areas of agreement. • Frame communications around strategic goals, belief systems, and areas of agreement. • Provide information that reduces fear. • Measure! Measure! Measure! • Trumpet successes. 18
- 20. Important Considerations Regarding Minimum Qualifications (MQs) • Represent the minimum competence level needed to perform the job. • MQs are often highly subjective. • MQs often result in adverse impact or less diversity in your candidate pool. • MQs sometimes screen out outstanding candidates. • Processing applications can be very time consuming – evaluate when in your process you should review applications. 20
- 22. Best Practices for Hiring Announcements • Emphasize EEO and diversity values. • Describe the application and hiring process. • Be specific about the job description and required qualifications. • Provide information on the benefits and available career paths. • Advertise through want ads, minority publications, television (including Cable/Local Access), and radio stations. 22
- 23. 23 How Applicants Learned About Fire Fighter Job Openings 42 40 36 36 39 23 4 15 7 15 7 12 0 5 10 15 20 25 30 35 40 45 50 Current Member Friend or Family Website Radio Called Dept. Want Ads White % Black %
- 24. Recruitment 24
- 25. Recruitment Best Practices • Involve women and minorities in the recruitment process. • Show diversity in ads brochures, recruiters, media coverage, etc. • Ensure that women and minorities in the department have a visible presence. • Encourage “word of mouth” recruiting among women and minorities. • Go where the women and minorities are! • Have a presence at cultural events and job fairs. • Build positive relationships with local minority community leaders and minority groups. • Provide mentoring and training/tutoring for the tests. 25
- 26. Targeted Recruiting Do’s & Don’ts • Do not exclude non-minorities from any of your recruitment initiatives or opportunities. • Work with leaders in the minority community to encourage minorities to take advantage of recruitment initiatives or opportunities. • Do not use quotas. • Be consistent and fair to every candidate in your hiring process. • Hire highly qualified candidates. 26
- 28. Best Practices for Making Decisions about Your Assessment Process • Don’t make assumptions – Investigate and Measure. • Recognize that there is no perfect process or perfect test. • Conduct a validation study or transportability study. • Be careful about making decisions based on single administrations or small samples. • Understand sampling error. 28
- 29. 29 Example: One Client’s AI Ratios Over Multiple Administrations 1.41 0.84 0.69 0.67 0.87 0.78 0.72 0.44 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1 2 3 4 5 6 7 8 AdverseImpactRatio Administrations Combined Sample = .75
- 30. Example – Simpson’s Paradox 30 Male Female Applicants Hired % Hired Applicants Hired % Hired Year 1 150 30 20% 40 16 40% Year 2 200 70 35% 50 35 70% Year 3 100 15 15% 50 15 30% Year 4 50 2 4% 600 48 8% Total 500 117 23% 740 114 15% Adapted from Bobko & Roth (2004)
- 31. 31 Evaluate Testing Medium Just as You Would Any Test • Research on the use of video or computer-based tests is inconsistent. • Research suggests that reduced AI may sometimes be the result of something other than the change in medium: – These tests are assumed to reduce AI by reducing the reading comprehension demands of a test. However, this can hurt validity if the acquisition of job knowledge or classroom training is required. – These formats may expand (or reduce) the domain of constructs being measured – Reduced AI may be result of reduced reliability. • Computer-based and video-based tests should be evaluated just like any other test—in terms of validity, reliability, AI, cost, and practicality.
- 32. Written Tests 32
- 33. Ask the Right Questions When Evaluating the Validity of Potential Tests • What does the test measure? Are those dimensions consistent with your job analysis results? • Does the test have a adequate validation report? • Was the test based on a job analysis of the position you are hiring for? • Is the recommended use for the test consistent with your process? • Where was the test validated? • Is the validation sample representative of your position? • What was the criterion used for the validation? • Does that criterion represent critical performance dimensions of your position? 33
- 34. Ask the Right Questions When Evaluating the Adverse Impact of Potential Tests • What was the size of the sample(s) used to evaluate adverse impact? • Was the sample similar to our applicant pool? • Is the adverse impact sample representative of our labor market? • Was adverse impact evaluated in an incumbent or applicant sample? • What are the adverse impact ratios for relevant protected groups at pass points you are likely to use? • Is the pass point at which adverse impact is reported similar to the pass point you will use? • What is the SD Difference between relevant protected groups. 34
- 35. 35 Measure a Broad Range of Skills Early in the Process • Some agencies administer a cognitive test in the first stage and measure non-cognitive skills in a later stage (e.g., interview). • This strategy will screen out many individuals who have strong non-cognitive skills and would make more complete candidates. • Measuring a broad range of skills can increase the validity (i.e., the quality of the candidate pool) and minimize the AI of your first stage (as well as your total process).
- 36. 36 Example—Which Candidate Would be the Best Hire? Cognitive Interpersonal Emotional Outlook Practical Candidate A 87 60 60 60 Candidate B 85 70 70 70 Candidate C 83 90 90 90
- 38. 38 Legal Background of PATs • Many Title VII standards have never been interpreted by the Supreme Court, and lower court decisions have been inconsistent and contradictory (Brooks, 2001). – “It is not an overstatement to say that no other aspect of the personnel selection process is fraught with more difficulty or clouded with more ambiguity than physical ability testing.” Larry Hoover, “Trends in Police Physical Ability Selection Testing” – A 1997 survey of court reviewed police and fire PATs found that less than 10% were successfully defended. – Most valid PATs will demonstrate at least some adverse impact with regard to Female candidates.
- 39. 39 PAT Recommendations • Use a PAT that utilizes job simulation exercises. – They tend to have lower adverse impact. – They tend to be highly correlated with physical job performance. – They are the easiest to defend legally as being job- related. • In addition to measuring strength, including events emphasizing balance, flexibility, and endurance can increase validity while reducing adverse impact. • Use a compensatory model. • Ensure that administrators and proctors understand that making changes to the test can invalidate the test.
- 40. More PAT Recommendations • Make sure individual events are supported by the job analysis. • Record data for individual events. • If women are underrepresented in the department, recruit women from colleges and gyms to run through the PAT before setting a cut score. • Communicate the importance of physical fitness to the candidates early in the process. • Show a video of your PAT to the candidates and put the video on your website. • Provide mentoring and practice to candidates. 40
- 42. Oral Interview Best Practices • Use Structured Interview format. • Ask questions that assess situational judgment and past behavior. • Create scoring benchmarks with Subject Matter Experts. • Use mock candidates to allow assessor panels to practice working as a team. • Use a multiple hurdle or compensatory model with written test scores. 42
- 44. Best Practices in Accommodating Disabilities • Require a letter from a doctor with a diagnosis and a recommended accommodation. • Establish a time frame for the letter from the Doctor. • Remember that accommodations need to be reasonable – the candidate must be able to reasonably perform the job. 44
- 46. Preference Points Considerations • Make sure your preference points don’t overwhelm the variability in your scores. • Measure the impact of your preference points on the distribution of your scores and your hiring process. – Do they prevent the top candidates from being hired? – Do they reduce the diversity at the top of your rank-order list? 46
- 48. Evaluate Your Background Check Criteria • There is very little research investigating the validity and adverse impact of many background check criteria. • There is very little consistency in the background check criteria used throughout the country. • Make sure you can defend the criteria you use. 48
- 49. 49 Example: Investigating Credit History During Background Check • EEOC has publicly urged organizations to avoid using credit history in making pre-employment screening decisions. • Credit history has substantial AI – Freddie Mac study found that the race-credit correlation is stronger than the income-credit correlation. • Bad credit histories are often the result of factors unrelated to job performance (such as income loss, medical problems, family breakup). • Women sometimes have poor credit histories due to ex-partners who have failed to live up to their legal obligations (child support, alimony).
- 50. 50 Using Credit History as a Disqualifier in Background Investigations Is Difficult to Defend in Most Cases • There is no evidence that credit history is related to job performance. • There is no evidence linking poor credit history to theft, fraud, or criminality. • There is no evidence to guide creating criteria for rejecting someone due to poor credit history (how much bad credit history is too much?) • Credit records are notoriously susceptible to errors.
- 52. Final Considerations • Remember that Rome was not built in a day – making changes may take a substantial amount of time. • Focus on the changes you can make. • Continue your professional development and stay aware of current research, litigation, and best practices. • Continue to measure!!! • Periodically re-evaluate your process. 52