Social Glue: Everything I learned about coding from Diego, a Harris's Hawk

Editor's Notes

• #3: I recently got to fulfill a childhood dream and the reason I was able to do it is pretty interesting. If you grew up dreaming of being a falconer, and who among us did not, you might know that the raptors used in traditional European falconry are all clinically insane, or at least so terrified of humans that they take a long time to acclimate and will usually only tolerate the presence of one or two specific handlers.   But this beauty here is not a traditional European falcon. It’s a Harris’s hawk, a native of the Americas from the Sonora desert down to Argentina and Chile. Unlike almost every other raptor, Harris’s hawks are cooperative hunters. They form packs of two to six, and they’ll scout ahead for one another, or they’ll split up so that some are flushing out prey and the others come in for the kill.
• #4: Now if you’ve ever read any of Jared Diamond’s books that touch on the domestication of animals, you’ll recognize this combination of a calm temperament and a pack mentality as a set of traits that really lends itself towards animals forming cooperative relationships with human beings. And sure enough, the Harris’s hawk has been used more and more in falconry since the 1980s, so much so that the definitive book on the topic is called the Harris’s Hawk Revolution. It’s because these hawks are are so social and friendly that an amateur like me can go out – I went with West Coast Falconry, near Sacramento – and have a hawk fly onto my glove. And while they’re American natives, they’ve become the most widely used hawks in the West. They’re used for pigeon control in Trafalgar Square. Two more cool stories about the sociability of the Harris’s hawk, as told to me by Master Falconer Jana Barkley. One wild pack was observed to have an adult male member with a deformed talon, and this guy became a stay-at-home dad. He’d care for the fledglings while the other birds brought back meat from the hunt. The other story, that really blew my mind. Not only can you hunt Harris’s hawks that are already bonded to each other; groups of falconers can bring their own Harris’s hawks that have never met before and form spontaneous packs. The hawks will figure out their own pack hierarchy and hunt together.
• #5: All in all, they’re the kind of keystone predator that, like us, benefits from a range of cooperative behaviors, like kindergartens and specialization. I especially like this quote for the way it illustrates that the hawks are putting their differences, not just their raw numbers, to good use.
• #6: Here they are, doing their thing. Obviously I’d like us to take some lessons from the Harris’s hawk like hey, maybe we should be highly intelligent and social too, but I don’t want to be too motherhood and apple pie about it. Problem-solving in real time – which is fundamentally what both hunting and programming are – is difficult any way you slice it. Let’s acknowledge the difficulty of what’s been undertaken here, and then look at a couple of approaches that I think look promising in how to address it.
• #7: Raw numbers are a mixed blessing in any case: I mentioned that the optimal cast size – the cast being the hunting group – is 2 to 6 members, which maps pretty precisely onto everyone’s rule-of-thumb for good software development or indeed any other team sizes.
• #8: We’ve known for forty years that the communications overhead involved in adding people to a project increases exponentially as the number of people involved increases arithmetically. In other words, the bigger the team, the less effective it is in working together at a common task. That’s why Fred Brooks called his classic book on the subject “The Mythical Man-Month.” Brooks worked at IBM where he managed the development of the OS/360 operating system. The project ran late, and he added more programmers, and in the end he concluded that his doing so had made the software ship even later. Brooks’ argument is that there’s no such thing as a man-month. You can’t speed up software projects by adding people any more than you can speed up a pregnancy by having nine women carry the baby for one month each.
• #9: You can actually calculate the communications overhead of adding people to projects as n squared minus n over 2, where n is the number of people and the answer is the number of links.
• #10: And when you plug that back into a guesstimate for optimal team size, to figure out where the productivity of the group isn’t being overwhelmed by the noisiness of the system, lo and behold, you get teams in the exact range of casts of Harris’s hawks: 2 to 6. It seems like this might be an actual hard physical limit of the universe or maybe just of carbon-based avian and mammalian brains, but a hard limit nonetheless. So here’s the question I have for you, my fellow attendees at Gluecon: can we break it?
• #11: Because that’s not the only assertion that Fred Brooks made in the Mythical Man-Month. The other thing he claimed is that complex programming projects don’t lend themselves to being tackled in discrete parts.
• #12: I didn’t want to take this at face value without checking with my guru, Edsger Dijkstra, who taught me everything I know about computer science. Dijkstra was a Dutch supergenius who in the 1970s and 1980s wrote a ton of short essays about all kinds of topics and circulated them with his colleagues and students – you might recognize the phrase “Goto Considered Harmful” - and these have all been collected and published online by the University of Texas. They’re amazing, I keep coming back to them and learning more every time. Dijkstra believed that programming was a branch of mathematics, that learning the bells and whistles of different programming languages was a distraction, and that the core job of a programmer was to sit in front of a blank sheet of paper and “ponder” until she reasoned herself towards an algorithm that would solve her problem.
• #13: So I reread an earlyish Dijkstra essay, EWD473, “On the teaching of programming, ie on the teaching of thinking,” where he tackles this question of modularization that Brooks has brought up and acknowledges that yeah, the communications overhead is a thing… unless your interfaces are sufficiently thin. And I thought, huh, what does he mean by that?
• #14: Here’s what he means. “Here, however, we see in all its clarity a quite different and more fundamental purpose of modularization, viz. to reduce the total reasoning (irrespective of whether thereafter it will be done by different groups in parallel, or by a single one in succession.)” Okay, so English was his second language, but what he’s saying here is really awesome. You can use that pondering step to think about how to break the problem up, and then, if you have good interfaces, you can parallelize your two to six person teams to work on really, really huge problems. Such as for example Web-scale IT.
• #15: I think what we’re seeing right now in Web-scale software engineering are a couple of very promising approaches for routing around the hard upper limit of team sizes. One approach is technical – let’s call it modularization, although I’m thinking specifically of the proliferation of services exposed through clean APIs. And, more recently, of the proliferation of containers, which we’re going to hear plenty more about from our next speaker, since he’s the person responsible. The other approach is cultural. Let’s call it devops, although again, that’s only an approximation. The argument I want to make to you today is that these two approaches are inextricably linked. It’s the combination of the two that I call social glue.
• #16: I was pretty sure you were going to want some practical examples, and ones that weren’t from the 1970s. Although I’m sure Flickr seems that long ago to some of you. Flickr started out as a company called Ludicorp, which built a virtual world called Game Neverending. It was the Second Life of its day. It failed, and was resurrected as Glitch, and failed again. But Flickr, the Instagram of its day, was originally built as a photo-sharing application for use within Game Neverending, and in case you’ve ever wondered, that’s why Flickr filenames use the extension .gne. I still really love Flickr, but to me the most influential thing that ever came out of Ludicorp was this Powerpoint presentation from the 2009 Velocity conference. Flickr was trying to solve a huge problem – the sheer scale of the site was enormous for its time. But Flickr was also trying to innovate in real time, and to do this its developers and its operations teams needed to cooperate in new ways.
• #17: What they did, essentially, was to come up with a new social contract that was partly instantiated in software and partly in culture. On the software side, they automated a whole bunch of routine processes and shared resources and exposed all of this infrastructure as a set of services to the developers. This, all by itself, was a really interesting and innovative thing to do at the time. Effectively, they took everything they’d learned about how to operate their site and distilled as much of it as they possibly could into code. This code was the stored wisdom of the ops team. But at the same time – and, I would argue, equally importantly – they set a bunch of ground rules about how people were going to interact with this infrastructure, and with each other. Instead of being traditional European falcons, hunting alone and clinically insane, the dev and the ops teams were going to behave like highly social, intelligent Harris’s hawks, able to solve problems cooperatively. And to do this they had to be respectful of one another, to place a lot of trust in each other – the Linux and Xen programmer Jeremy Fitzhardinge has called trust a cache for reducing cognitive load – to cultivate a healthy attitude towards the inevitable disasters that were going to happen, and to not blame one another when those disasters did indeed take place. If code is stored wisdom, culture creates a safe space in which ideas can surface from the group and be chosen on their merits. That’s what it means for trust to reduce cognitive load. It means clearing a space in which people can sit down and ponder and try to find solutions for really difficult problems. This actually makes me think about what a company is. A company, especially a tech company, isn’t just a legal fiction with a tax haven in the Bahamas. It’s not just one idea or one technology. If it were, companies that pivot would be some kind of zombie ghost. No. A company is a set of companions. It’s a group of minds that solve problems better together than they would if they were apart.
• #18: This should all go a long way towards explaining why I am currently completely obsessed with Netflix. Culture is such a crucial competitive differentiator to Netflix that the most recent version of its Freedom and Responsibility deck runs to 126 slides. Don’t worry, I read it so you don’t have to (although you probably should.)
• #19: This is the summary slide, and while some of these are a little too Who here likes puppies? for my taste – which companies don’t want high performance, for example – some of them are worth thinking about in more detail. Point three, freedom and responsibility itself, is directly descended from the respect and culture points in the Flickr 10 Deploys per Day deck. Netflix, like Flickr, exposes an ungodly amount of its infrastructure to all its employees, providing them with huge amounts of transparency and agency. In return for placing this massive amount of trust in its people, Netflix expects them to use these resources in a responsible way. Context, not control, is another super-interesting one, because part of what we’re talking about here is moving from what Ori Brafman in his book The Starfish and the Spider called a coercive, command-and-control type of organization into a more open one, with a flatter hierarchy. I’ve bet my career on the belief that open organizations can outcompete coercive ones. The third point I’d like to tease out here is the one about being highly aligned but loosely coupled. Netflix positions this as the best of both worlds between a monolithic, command-and-control organization and an open organization where every unit is behaving too independently to ever get anything constructive done. Netflix teams have a very high degree of autonomy, but they all seek to be closely aligned in working towards a common goal. And this applies within teams as well, so that although each individual is working on his or her own piece of the puzzle, the team is able to put together a service and expose it through an interface such that the interface functions as the team speaking with a singular voice.
• #20: That’s the best practice for pulling off a highly modularized, large-scale project, and the infrastructure that Netflix has managed to implement this way is nothing short of stunning. You probably know that Netflix runs its infrastructure on AWS, and you may know that Netflix has open-sourced a bunch of the components that it uses to deploy and run that infrastructure. But you may not realize just how rich and complex an ecosystem the company has grown around every aspect of this undertaking. From monitoring and assuring availability –
• #21: To managing the cloud –
• #22: To building new applications –
• #23: To maintaining state –
• #24: To sharing infrastructure –
• #25: To developer productivity –
• #26: To harvesting customer data. Every one of those projects I just showed you is an open-source module with a clean interface that can be used with or without the rest of the Netflix environment. There’s already a third-party ecosystem starting to build applications on top of all this. And when you stop and think about it for a minute, and remember that code is stored wisdom, is a way to capture the algorithmic solution to a problem that is the result of serious and sustained pondering, you start to appreciate just what an extraordinarily powerful and productive team Netflix has managed to put together, thanks to its ironclad commitment to creating a culture that lets people solve problems and store their wisdom in a safe and supported way.
• #27: As a society, we face massive, unprecedented problems, not just around storing and streaming photos and films but around energy and food security, education and health care. I talk to the CIOs of some of the world’s largest enterprises every week, and they’re all engaged in replatforming efforts so huge that I can only describe them as efforts to reboot the enterprise.
• #28: As you can see, where the old enterprise worked off real-world models, the new enterprise simply wouldn’t exist without access to vast computing resources.
• #29: And when you go through what’s on the CIO’s mind, the same themes come up again and again. This particular list comes from the CTO of a running shoe company, but I hear exactly the same concerns from executives in the financial services industry, in health care, in manufacturing, retail, e-commerce, consulting, pharmaceutical – you name the sector. Starbucks processed a billion dollars in transactions off its mobile app last year. That’s a pretty strong argument for moving away from waterfall-style development, and it’s illustrative of the way every single enterprise I talk to is now thinking of itself as a software and information company.
• #30: At no time is the ability to store wisdom for later retrieval more critical than it is in a crisis. Enterprises aren’t just writing software to create mobile apps. They’re writing infrastructure automation to cope with the inevitable failure of the hardware supporting those mobile apps.
• #31: And the enterprise’s reinvention of itself as a software company is a huge part of what’s driving this profound shift from proprietary to open source software. If you buy proprietary enterprise software from a vendor, you are in a client relationship to that vendor. If you build your own infrastructure from open source components, you own your destiny.
• #32: That’s not to say that you want to own and manage everything yourself. Enterprises need to own and manage what’s differentiated for them. For everything else, there’s SaaS.
• #33: . Either way, private IT can never outcompete the public cloud in economies of scale, which is a fundamental driver pushing more and more enterprise functions into the cloud.
• #34: And we’ve already touched on this in the Starbucks example, but the mobile revolution has already happened. I get as much of my own work done on my Android phone as I do on my laptop and desktop I depend on them all synching nicely, and my colleague Al Tsang at Strongloop is giving a talk tomorrow on how to build apps mobile apps that play nice with others. What’s most interesting to me about the proliferation of smart phones and tablets is that they’re preparing the way for an even larger deluge, two or three orders of magnitude larger – the Internet of things. Mobile is trainer wheels for an Internet adapting to unprecedented scale.
• #35: And the techniques that serve us best as we try to adapt our infrastructure are to move away from silos and into a world of services, APIs and containers. My contention is that as good as the technology is, we’re never going to get there without an equally good culture.
• #36: The rarest and most precious commodity on earth is the ability to solve such massive problems. These problems are way beyond the capacity of a single person to solve, so it’s not going to be a psychotic European falcon like a Steve Jobs that solves them for us. Collective intelligence isn’t strongly correlated with the intelligence of the smartest person in the room. It is much more strongly correlated with a culture of respect, trust and avoiding blame, a culture that allows real wisdom to emerge from the group.
• #37: So that’s what I mean when I talk about social glue. Code stores wisdom by encapsulating a whole bunch of lived experience and the associated pondering on how to solve really difficult problems. To get at that lived experience and to empower people to stare at a blank piece of paper until they figure out how to solve the problem, you have to foster a culture that rewards risk-taking and doesn’t punish failure, a culture that values the contribution of all its responsible members. And the well-formed interfaces are both the machine-readable APIs, and the fundamental assumption of goodwill in the way we treat one another. That fundamental assumption of goodwill is what allows Harris’s hawks to hunt cooperatively with hawks that they’ve never met before, and with humans that they haven’t met before as well. It is a superpower. I think the real, world-historical significance of the movements I’m trying to describe here are that we have in our hands – you, my fellow Gluecon attendees, have in your hands – some technical and cultural tools of immense, unprecedented problem-solving power. Although it may not feel like it, by virtue of being in this room, you are helping to create the future. What kind of future would you like it to be? Who would you like to see included in it? Whose ideas are important, and whose wisdom is being overlooked? Why? You will be astonished at how the choices you make today about what kind of software you want to write and what kind of company you want yours to be will reverberate down the whole length of your career.
• #38: I’ll close with a quote from the late, great Jon Postel, whose words on Internet communications protocols are applicable to a much, much broader range of human interactions: “Be liberal in what you accept, and conservative in what you send.”
• #39: Here he is at last, my good friend Don Diego Alejandro Inigo Montoya del Gato, being admirably liberal in what he accepts and fundamentally assuming my goodwill. Thank you very much for your time and attention, and have a wonderful conference.