Distributed systems are hard; distributed systems of people are harder

Editor's Notes

• #2: Here to speak to you about the lessons we've learned at Chef being a "remote-first" company -- how we got there, how it's gone so far, and where we're continuing to improve.
• #3: Chef is a very distributed company – has been right from the very beginning, even though it was founded in Seattle. We wanted to hire the very best engineers no matter where they were located. In about 2012-2013 when I started, Adam Jacob, Chef’s founder, moved to San Francisco so that his wife Katie could take a job with change.org So that introduced us to the concept of distributed leadership, and we’ve not really looked back. People ask me all the time, isn’t it tough to be a remote product manager? And it can be challenging – but it’s a different set of challenges from working on a team in an office. It also affords me great flexibility in my schedule.
• #4: Chef being a remote-first company has afforded me great personal flexibility. Actually from just “down the road” and I moved to New York City in 2011 to start a graduate degree. After that, I worked in a traditional office job as an operations engineer. Chef’s my first fully remote – or distributed, as I like to say, more on that later – job, where I’ve worked for the last 4.5 years, first as a traveling consultant and for the last 2.5, as a product manager. It’s allowed my wife & I flexibility to move to another city, so now I live in Portland, Oregon – even though Chef is based up in Seattle. One of the reasons I specifically chose to give this talk here in Detroit – it’s the first time I’ve given this talk, so this is all fresh material – is because I’ve been able to watch, from a distance, what’s happened to Detroit. But if I had any money to spare, I’d be investing in Detroit because I think the city has great potential. So thank you all for coming here, and organizing DevOpsDays Detroit, because of this exact reason.
• #5: ... well, let me ask you this. Who here is familiar with this book, from the folks at 37 Signals? I like this book, but like most business books, it suffers from the flaw of only telling you the happy path. “37 Signals is a magical company where everyone works remotely and it’s all sunshine & unicorns”, right? Who the hell wants to hear that. What I’m here to give you is practical advice for making distributed teams work – there is good, there is bad, there are compensating measures for the bad. I want to share with you our experiences doing this at Chef as we’ve grown the company from 50 to about 300 folks now & give you some actionable things you can take away for your distributed teams. Overall, I am bullish on distributed teams; the benefits outweigh the costs. And I think distributed work has a huge part to play in revitalizing cities like Detroit.
• #6: Words are important so I want to bring this up right away. Distributed team – all employees are first-class citizens and the tools & culture support that. For example... you don’t just get a nice desk and chair if you work at “head office”. (Chef gives remote employees a one-time stipend up to $1000 to spend on office furniture.) At all-hands meetings, you don’t get stuck on a dial-in with shitty audio while the CEO talks to the team in-person at HQ with fancy drinks; all-hands meetings are conducted 100% online with the CEO on Zoom from his/her house. “Remote employees” can often be akin to ”onshore outsourcing”. I once went to see a customer at a bank and they were talking about their co-worker Bob, whom none of them had seen face-to-face in two or three years. Two or three years!
• #8: This is the first and last Rust joke I’ll make in this presentation... Distributed teams have a lot in common with distributed systems. Actually you could learn a lot about what makes strong distributed teams by thinking about what make good distributed systems. So let’s start there...
• #9: Eric Brewer: it is impossible for a distributed data store to simultaneously provide more than two out of the following three guarantees: Consistency. Availability. Partition tolerance.
• #10: We’re not building a distributed system to serve requests... so we can give up availability. Actually we must do so: timezones appointments personal time life events vacation
• #12: Vision
• #14: Amazon does this too but with different artifacts. Write press release first
• #16: Say more about passive-aggressive comms Domain-driven design
• #19: How many people use Slack these days? How many of you like Slack? How many of you hate it? Why?
• #23: From 10 years ago! Replace with the pic of me with the new glasses
• #26: Leaf: Make the decision, act on it, do not report the action you took. Branch: Make the decision, act on it, report the action you took daily/weekly/monthly Trunk: Make the decision. Report your decision before you take action. Root: Make the decision jointly, with input from many people. These are the decisions that, if poorly made and implemented, could cause major harm in your organization.