A Successful Hiring Process for Data Scientists
Download as PPTX, PDF0 likes67 views
The document outlines the structure and process for interviewing data science candidates, focusing on their ability to learn and adapt to new algorithms and project requirements. It emphasizes integrating hands-on projects and reasoning questions into interviews to better assess candidates' problem-solving skills and fit for the company. The aim is to ensure candidates align with the company's objectives while fostering a mutual understanding of the job's expectations.
A Successful Hiring Process for Data Scientists
- 1. KDD 2022, W26 Half Day Workshop – Tuesday August 16 3rd IADSS Workshop on Data Science Standards – Hiring, Assessing and Upskilling Data Science Talent IADSS link https://www.iadss.org/ Workshop link https://www.iadss.org/kdd2022 One Hiring Process – Quiz for Learning and Reasoning Greg Makowski Head of Data Science Solutions greg.Makowski@jci.com www.LinkedIn.com/in/GregMakowski Building Technologies & Solutions
- 2. 2 © Copyright Johnson Controls. All rights reserved. Johnson Controls—Public. Any unauthorized use, copying or distribution is strictly prohibited. Johnson Controls Confidential Context My background • I have been deploying data mining and data science models since 1992 • I have been hiring and building DS teams since 2010 Assumptions driving requirements and design • You can have candidates describe their past models deployed and development process. That is not hard. – However, that may not be a close match for upcoming work • The types of future algorithms and architectures will likely be constantly changing, between the project requirements and what is developing as State Of The Art (SOTA) • Therefore, it would be good to get a sense of how a candidate would do on future, upcoming projects and new algorithms • There is a large variety of new things to learn, of new algorithms or tweaks at a conference – but you shouldn’t expect the candidate to know all • Expect that constant learning is needed in the future • interview for the candidates learning process
- 3. 3 © Copyright Johnson Controls. All rights reserved. Johnson Controls—Public. Any unauthorized use, copying or distribution is strictly prohibited. Johnson Controls Confidential Context – other parts of the interview process • Job description – looking for a person that has ”deployed 3+ models”, like a ”medical residency”. 3-12 yrs • Other parts of the interview process already cover common basics – basic coding or software development skills – algorithm skills on algorithms they know well or have deployed • Treat the interview process as a “dating game” to get a good match for the candidate as well as the employer – What the company has to offer, existing enterprise software, tools, past projects – Career path in Data Science – “A day in the life of”, a “week in the life of” the candidate in the role – Personal responsibility, ownership, how the they can contribute to the employers OKR’s (Objectives and Key Results) – Intellectual variety – vertical markets, customer personas, algorithm families – Opportunities to intellectually grow, learn and adapt • In final round, have candidate present on past project for 45 min, which includes Q&A – Focus on their DS “value add”, avoid anything confidential to past client or co – Like an internal DS presentation or to DS contacts at a client site, to make it easier – Can be a reused conference or meetup presentation – A chance for all the DS staff in a compatible time zone to understand the past project and ask questions Exciting Mission • Save the world • Save people • Save machines
- 4. 4 © Copyright Johnson Controls. All rights reserved. Johnson Controls—Public. Any unauthorized use, copying or distribution is strictly prohibited. Johnson Controls Confidential Design of one part of the DS phone interview process ”Teach” for 15 minutes, going over content in slides • Let the candidate ask questions, it is fine to ask to go back and forth in the slides • Give a problem context, a client role in a vertical domain, their values and KPIs • Pick something that goes deep into an algorithm, not at the “surface level” of a Python function call, but on how the algorithm interacts with the data • Every few slides, ask if they understand or have questions • Describe in terms of different modalities (math, geometry, process) “ask reasoning questions” that may take 15-30 min
- 5. 5 © Copyright Johnson Controls. All rights reserved. Johnson Controls—Public. Any unauthorized use, copying or distribution is strictly prohibited. Johnson Controls Confidential Design of one part of the DS phone interview process “ask reasoning questions” that may take 15-30 min • One question is more algorithmic or geometrical • One question is more about “fiduciary responsibility to the values and KPIs of the client”, for the DS to think as a consultant or in the business deployment terms • Tell the candidate they can ask to go back and forth over any of the training slides • They can ask questions of the client contact. Then roleplay as a client (not knowing DS). • Track the time it takes per question, as one metric to compare candidates. It is mostly used for comparisons of large differences • For any ”binary” answers, the candidate has to explain “why”. Ask probing questions • Ask the candidate to declare a “final answer state”. Don’t stop them if you hear something close. They should be confident and describe their reasoning – When deploying in the future, they have to be confident to know their “final answer state”
- 6. 6 © Copyright Johnson Controls. All rights reserved. Johnson Controls—Public. Any unauthorized use, copying or distribution is strictly prohibited. Johnson Controls Confidential Design of one part of the “on-site” interview process Go over a short 2 week POC project in slides (something I know well and worked on). Provide background slides w text and diagrams for 15 min • Client requirements • Data, it’s noise, what it represents in terms of a physical process or human decisions, sample tables of different structures • Initial EDA that you could take for granted that most people would do • Offer to “role play” or be a “holodeck simulator” to answer questions – Of the client, who knows objectives and data at a business process level – Of the data, for a specified EDA, program or model, get a high level answer – A fellow DS on the team, who doesn’t know the problem – may trigger questions – Can go back and forth to any of the background slides • Questions for ~30 min: – How to structure the data to fit some algorithm family (very involved for this project) – How to train the models, model development process (less time, maybe 10 min) – Once you have an algorithm result, so what? What business process do you change to get value or a business KPI metric? How to estimate the value? (not obvious for this project)
- 7. 7 © Copyright Johnson Controls. All rights reserved. Johnson Controls—Public. Any unauthorized use, copying or distribution is strictly prohibited. Johnson Controls Confidential When designing questions, consider: Pick an algorithm or algorithm feature that is intentionally obscure • Want to see a candidate learning and adapting experience, how they ask questions • Want to avoid 1 in 10 candidates who already know the answer – then you can’t compare how they “would learn X the first time” In the age of remote interviews, a person can web search easily • Check common web searches in advance, what they return and how that could be used. Consider how you may tweak your question or setup • Maybe don’t use an algorithm or feature name, just call it “alg X”
- 8. 8 © Copyright Johnson Controls. All rights reserved. Johnson Controls—Public. Any unauthorized use, copying or distribution is strictly prohibited. Johnson Controls Confidential Questions, discussions, alternate approaches, advice? Since 2010, candidates hired this way have been good at • adapting to new projects and requirements • No retention surprises or mis-hires