Lesson 3: Hiring

(This is a follow-up to my ‘0. Lessons from a career in tech’ - see comments for a link)

Hiring processes in large companies tend (in my experience) to be very well structured. But in small or medium sized businesses much less so. While there are some good reasons why a small startup might see fewer benefits from a strongly structured process, in medium sized businesses (where you are likely hiring repeatedly into the same role description: software engineer, data scientist, etc), this is not true. A good process maximises your chances of identifying and successfully hiring the right people, AND it minimises your chances of making a bad hire.  And many of the same lessons can and should be applied when you are trying to hire just a single person into a specific key role.

I’ve certainly made a few bad hires in my time, to be clear, and I made sure to learn from each one. Fairly early in my career I learned how to effectively and repeatedly hire many people into standardised roles, it took me a little longer to do that really effectively for single roles.

Hopefully you work somewhere that has good hiring processes and you won’t learn much from what I write, but if not, do keep reading….

While I’ve continued to learn through hiring for many different role types throughout my career (engineer, PM, data scientist,…), my biggest learnings were when I worked at King, where I built a function of about 150 data scientists, from scratch, over a few years.

Hiring well requires clarity on a few things.  Perhaps most important: in each of the areas where you want the person to show some capability, what is the bar you are looking for?  Obviously you are looking for the “best possible people”, but if you imagine hiring 100 people, then there will be a minimum bar below which you won’t hire, some kind of average across all areas that they need to reach, and then a small number of critical factors.  The critical factors may depend on what your company needs at that time, and your product, but should also depend upon your views of what is more vs less teachable.  Capabilities that can be learned quickly can have a lower bar than those which are harder to learn.  The hardest to learn will be these critical factors - non-negotiable bars the candidate must hit (ie convincingly demonstrate in interview).

It is essential to identify these different capability areas up front, and classify them (critical vs the rest), and define a scale or threshold for each. Then you can define an interview loop accordingly (and your colleagues on that loop need to understand and agree with your process!). The interview loop must give you a clear yes/no on the small number of critical factors, and some kind of grading across each area. If a candidate doesn’t hit a critical factor then that is an immediate “no” irrespective of everything else. This bears repeating: no matter how amazing the candidate is in other areas, if they fail on a critical factor, they are not a match and must not be hired.

For example, at King we had two critical factors for individual-contributor (i.e. non-manager) Data Scientists:

Factor 1: Could the candidate clearly explain some technical topic in which they claimed expertise?  And when probed more deeply on aspects of that topic, could they be clear about what they knew vs didn’t know.  Could we get an indicator on their ability to quickly learn new technical things?

Factor 2: Could they reason (intuitively - NOT technically) and communicate about typical things we observed in our games - retention curves, for example.  In other words could they apply their analytical mind to practical, applied things and generate hypotheses, explain how they might prove/disprove those, etc.  Could they put themselves equally well into the mind of a player and the mind of the company that builds the game?

Design the interview process so that each interview assesses just 1-2 capability areas. Each interviewer needs to be focused just on their part of the process (so tell them or train them in what you expect).  Each of the two critical factors had it’s own interview (I personally conducted almost every ‘Factor 2’ interview for 5 years - probably close to 500 interviews!).

There were candidates with incredible technical depth and skills who absolutely failed on Factor 2 above.  We never hired them.  Once there was someone so exceptional in other areas that even though they failed this interview (terribly), the hiring manager insisted on bringing them in as a temporary contractor using their budget (side-stepping my blocking decision for permanent employees).  They didn’t even last 3 months.  A critical factor is critical for good reasons.  Don’t bend your rules!

Be on the candidate's side: Another very important dimension is how to get the best out of candidates?  You want to be on the candidate’s side and give them their best possible opportunity to shine.  Two thoughts on this:

a) Share clear details with the candidate - what capabilities will be assessed in each interview, so they can prepare accordingly.  You want them to be prepared and show their best! While being able to think on your feet is a useful skill, in most roles it is more important to be able to think carefully and prepare in advance - so give your candidates that opportunity.

b) In many interviews, you will reach an opinion quite quickly (e.g. 10-15 minutes into a half hour interview). Use the rest of the interview to try to probe and disprove your opinion.  If you felt they messed up and are a weak candidate, see what questions you can ask which might reverse that opinion? If they seem amazing, what questions could you ask which might reverse that opinion?

Single, specific, difficult positions:

While a lot of what I’ve written above is in the context of hiring for a large number of similar Data Scientist positions, most of the same lessons apply when you are just looking to fill a single specific, difficult position.  As you proceed through your candidates (with screening calls, followed by interview loops, etc), obviously you are in this case looking for the “best possible person”, but it is comparatively rare that you find yourself in a position with, say, two great candidates (who have passed all the interviews) and you have to make a decision between them.  Great situation to be in (!), when it happens, but rare.

For difficult roles, it is more likely that you’ve rejected a few people, and are now in a situation where there’s a candidate who has finally passed all of the interviews, but with a few concerns.  Should you hire them?  There is always a strong temptation to try to fill the role, make the offer, and move on.

Here is where you need to be very honest about your criteria - and why it helps so much to have defined those carefully up front.  What is the bar they need to hit in each area?  What are the critical factors? Can you dispassionately rate the candidate against those, and conclude that they are, perhaps, not the right person?  Did each interview assess the candidate against the right capability bar?  If this has been clarified up front and built into the interview loop, things are much simpler, but otherwise it can be a tricky situation. This indicates that your interview process was not good enough, so diagnose what went wrong and improve it!

If you really, really feel the need to have the candidate take on one more interview, to resolve a tricky decision, then apologise to them, and do that. But if that happens often (or you need them to take on multiple further interviews), then you and your organisation have a problem. And great candidates will notice that and act accordingly. So you’ll only hire mediocre people that way, anyway!

I’ve certainly made a few wrong hires over the years, and some of them were at least partially caused by taking the tempting easy way out.  There are no magic answers to such situations, however as my career progressed I became better at making good decisions in such cases - and certainly became more accepting of the hard decision to keep looking. The time ‘lost’ keeping on looking will be much less than the time lost dealing with a wrong hire.