The Impact of Generative AI-Powered Code Generation Tools on Software Engineer Hiring: Recruiters’ Experiences, Perceptions, and Strategies
0 likes95 views
The study investigates the effects of generative AI-powered code generation tools on software engineer hiring, focusing on recruiters' experiences and perceptions. It finds that while many recruiters are familiar with AI tools, a significant portion has not adapted their screening processes to include these technologies, raising concerns about evaluating candidates' true coding skills. Recommendations include updating hiring practices and integrating AI tools into computer science curricula to enhance industry relevance and productivity.
The Impact of Generative AI-Powered Code Generation Tools on Software Engineer Hiring: Recruiters’ Experiences, Perceptions, and Strategies
- 1. The Impact of Generative AI-Powered Code Generation Tools on Software Engineer Hiring: Recruiters’ Experiences, Perceptions, and Strategies Alyssia Chen, Timothy Huo, Yunhee Nam, Daniel Port, Anthony Peruma
- 3. GenAI-Powered Code Generation ◆ Rapid advancement of Generative AI tools ◆ Transformation of software development practices ◆ Enhanced developer productivity ◆ Gap in research on hiring impact ◆ Challenges in evaluating true abilities
- 4. Related Work ◆ Most studies involve GenAI use in an educational setting or with developers ◆ Recruiters and students prioritize certain resume items differently and spent varying amounts of time in sections of the resume - Petersheim et al. (2023) ◆ Significant preferences for strong technical skills in the hiring process - Adnin et al. (2022) ◆ Significant increase in the demand for AI skills in most industries and occupations in the U.S. labor market from 2010 to 2019 - Alekseeva et al. (2021)
- 5. Goals & Research Questions Research Questions ◆ RQ1: Recruiter familiarity with AI tools ◆ RQ2: Adaptation of screening processes ◆ RQ3: Value of candidate AI tool experience ◆ RQ4: Importance of AI tools in CS curricula Goals ◆ Investigate how organizations adapt hiring strategies ◆ Understand recruiter perceptions of AI tools ◆ Examine evaluation challenges ◆ Explore integration with education
- 7. Data Collection ◆ Survey of 32 industry professionals ◆ Spring 2024 Career Fair ○ 30 organizations ◆ 24 survey questions ○ Focus on recruiter experiences and perceptions ○ Examined adaptation of hiring practices ○ Explored integration with CS education
- 8. Thematic Analysis To ensure reliability, three authors independently reviewed and categorized the responses, discussed any discrepancies, and reached a consensus.
- 9. 03 Results
- 10. Demographics (32 respondents) Male 17 (53.13%) Female 14 (43.75%) ● 13 (40.63%) reported “HR/Talent Acquisition/Hiring Manager/ Or Similar” ● 5 reported “Software Engineering/Software Architect/ Or Similar” ● 7 reported “Team Lead/Project Manager/Account Manager/ Or Similar” ● 7 reported “Other” 25-34 37.5% 35-44 28.1% 45-54 9.4% 55-64 18.8% 18-24 6.3% Gender Age Years of Experience Organization Type
- 11. Most common combinations: ● ChatGPT + Amazon CodeWhisperer ● ChatGPT + Replit AI ● Github Copilot + Replit AI RQ1: Recruiter familiarity with AI tools Which of these AI code-generation tools are you familiar with? ChatGPT 28 60.87% Github Copilot 8 17.39% None of the above 4 8.70% Amazon CodeWhisperer 2 4.35% Replit AI 2 4.35% Others: 2 4.35% Frequency of AI code generation tool usage for work
- 12. RQ2: Adaptation of screening processes No 65.63% Not Sure 18.751% Yes 15.63% Official guidelines or policies related to GenAI code generation tools when evaluating candidates No 17 (62.96%) N/A 6 (22.22%) Yes 4 (14.81%) “Provide a summary of your evaluation criteria”: ● Standardized Evaluation Criteria ● Verbal Discussion Have you changed the criteria you utilize to evaluate a candidate’s coding skills due to the wide availability of AI code-generation skills? Have you allowed candidates to utilize AI code- generation tools in a technical interview? Yes No 1 20
- 13. RQ2: Adaptation of screening processes Should candidates be allowed to utilize AI code generation tools during technical interviews? AI IS A TOOL (3) PROMOTES FAIRNESS (1) DEMONSTRATES PROFICIENCY OF CURRENT TECHNOLOGIES (1) YES NO CHALLENGE EVALUATING CANDIDATES TRUE TECHNICAL SKILLS (4) CONCERNS OVER QUALITY OF AI- GENERATED CODE (1) CHALLENGE EVALUATING CANDIDATES COGNITIVE SKILLS (2) Perceived challenges in assessing candidate skills with AI code generation tools Not sure Not hard Slightly Harder Moderately Yes No Not Sure 5 7 8 N/A = 4
- 14. Critical skills candidates should have to effectively use AI code generation tools: 1. Critical Thinking & Problem-Solving Skills (5) 2. Prompt Engineering (5) 3. Evaluate AI Response (6) 4. Familiarity of AI Tools (2) 5. Fundamental Computer Science Knowledge (1) RQ3: Value of candidate AI tool experience Frequency of asking candidates about their experience and skills in using AI code gen tools Extent of preference given to candidates who demonstrate skills in using AI code gen tools No preference Moderate Strong
- 15. RQ4: Importance of AI tools in CS curricula Importance of integrating AI code generation tools into computer science curricula Benefits: 1. Improved Productivity (5) 2. Equitable Access (4) 3. General Awareness & Preparedness (10) 4. Industry Relevance and Preparedness (3) Risks 1. Academic Dishonesty (6) 2. Challenges for Educators (4) 3. Student Overreliance (6) 4. Lack of Responsible Use (4)
- 16. 04 Conclusion
- 17. Threats Qualitative data may be limited Rapidly evolving nature of AI Limited Participants Participants come from various industries and are anonymous Conduct in-depth interviews or focus groups in the future Valuable insights at this specific point in time
- 18. Recommendations Industry-academia partnerships Focus on cognitive skills assessment Update hiring practices Integrate AI tools in CS curriculum
- 19. THANK YOU! Alyssia Chen Timothy Huo Yunhee Nam Daniel Port Anthony Peruma