2011 07 Living without your Linemen—OSCON

Editor's Notes

• #2: Abstract:\nIf a website architect is the quarterback, then site operations is the offensive line—overworked, underappreciated, and only noticed when it fails. They make you look good. However, four years ago cloud computing networks like Amazon Web Services and Slicehost have appeared. While deficiencies in frameworks in other languages have forced those worlds to adopt Infrastructure-as-a-Service, the PHP world—with it’s ultra-cheap shared-hosting (on one end) and tradition of dominance on some of the most trafficked websites (on the other)—has been slow to move. But as the technology continues to disrupt, modern web engineers will be expected to use their programming skills to not only build, but also provision and maintain fast, scalable websites.\n\nThe efficiencies of a web-based language and experience in scalable website architecture offer a unique opportunity for programmers to transfer their skills when wearing a sysop hat. Not to mention some of the best libraries for programming them are written in PHP! When going from a small pet project to a go-live site, maybe we can learn to live without our linemen.\n\n
• #3: The need for rich sites\n
• #4: I meant to say, “When I created the website.” Geez! One slip of the tougue is like hanging chad in your mouth.\n
• #5: \n
• #6: Because WordPress started before IaaS existed, Automattic runs systems in a more standard manner. Instead this is the experiences I have advising friends startups. (If you want to see how Automattic runs, talk to Barry Abrahamson. My extent of Automattic’s structure stops at, “Barry! I think I broke something!”)\n
• #7: \n
• #8: I DON’T think I can do your job, like some of the people you’ve worked with. ;-)\n
• #9: I’m a PHP Programmer. At ZendCon, a PHP conference, in 2006, the host made these.\n\nhttp://www.flickr.com/photos/calevans/285999927/in/set-72157594355906012/\n
• #10: I’m a PHP Programmer. At ZendCon, a PHP conference, in 2006, the host made these.\n\nhttp://www.flickr.com/photos/calevans/285999927/in/set-72157594355906012/\n
• #11: And it occurred to me when someone puts your face on a deck of cards you must be a terrorist. (I guess he put these so that the Java Strike Force could hunt us down and kill us.) So, if some of what I say shocks you or seems a little extreme, you’ve been warned!\n\nhttp://www.flickr.com/photos/papernapkin/740945436/\n
• #12: Let me explain a running theme: during a presentation at a PHP conference a number of years ago I said this… I meant to say, “When I created the website.” Geez! One slip of the tongue is hanging chad in your mouth.\n
• #13: That inspired some in the community to put up this website (the PHP IRC channel robot also does this). Clearly, according to phpdoc (and the #php-* IRC robots), I am awesome. :-)\n
• #14: Here is another. Apparently I didn’t need to fly to get to Portland… (That’s enough Keynote transitions).\n
• #15: Start with the opening keynote. Three takeaways\n1) Lots on discussion of the performance of PHP: xdebug/cachegrind, inclued, HipHop analyser\n2) Diagram of a modern web stack\n3) Architecture considerations and deployment strategies\n
• #16: Let’s preface each section with an anecdote. This one concerns an article I wrote during Christmas of 2009.\n
• #17: A few startups ago, I worked with a guy who was a better programmer than any of us.\n\nOne day, we got into an argument over a piece of open source import code — written in Python — that he had ported to C++. He had just finished telling me how much faster he had made it, when I asked, “What’s the point in that? Now that you’ve rewritten it, you own the maintenance of it. There is no evidence this code is even the bottleneck.” \n\nAhmdahl's Law\n\nThe point? The code was crap, and he had fixed it — the massive improvement in efficiency was an added bonus.\n\n“Look you’re right. It’s true I prefer to use a crappy, ugly, underperformant language like PHP, and you crank out C++ like John Henry drives steel. But, while you’re busting code with a hammer in each hand, I’m the guy with the steam-powered jackhammer. Sure, you win, but your heart will burst, and you’ll die.”\n\n“And here’s the thing,” I added with a devilish grin, “There’s only one you; there are a lot of people like me out there.”\n\nImage from: http://www.postalmuseum.si.edu/artofthestamp/subpage%20table%20images/artwork/legends/John%20Henry/BIGjohnhenry.htm\n
• #18: My friend said. “J2EE programmers only write five lines of code a day.” He’s a Java programmer so he added “They’re give REALLY GOOD lines.”\n\nImage: http://www.flickr.com/photos/tychay/1388234558/\n
• #19: Here was a popular book from 2008. One interesting point Malcolm Gladwell makes is that it takes 10,000 hours of diligent practice to achieve mastery in nearly anything of worth. Pretty soon everyone is tweeting, all I have to do is spend 10k ours on it and I’ll master it.\n\nImage: http://www.flickr.com/photos/tychay/1388234558/\n
• #20: But let’s break that down. Gladwell stole this from this book, but he deliberately misrepresents how much 10k hours is. Also, not just any practice (diligent practice is hard!). He makes 10k hours sound achievable, but it isn’t—its 10 years of work doing somehing nobody, even masters like Bobby Fisher, Tiger Woods, YoYo Ma find unenjoyable.\n\nImage: http://www.flickr.com/photos/tychay/1388234558/\n
• #21: Similarly let’s look at what 1500 lines of code/day means. The Net result is I think C++ is crap because it only does 3 lines of (work) assembly for every line of code. PHP does much more (less efficiently).\n
• #22: Either choose what you spend to spend your time coding wisely or get a lot of people with chainsaws\n
• #23: \n
• #24: This is an analogy a co-worker once told me.\n
• #25: “You’re the architect, so you’re like the quarterback”\n
• #26: “I’m a programmer so I’m like the Running Back\n
• #27: …or wide Receiver.” I guess the difference between a Wide Receiver and a Running Back is one is back-end engineering and the other is front-end engineering. (Which is which? I don’t know.)\n
• #28: And Systems is like the big burley guys up front. They make you look good.\n\nYou don’t notice them until the site goes down (get a holding penalty)\n\nHow many times have I had to field questions about “what do these people do? I dont’ see them working?” And I have to explain, “You don’t want them to look like they’re working. If they do, something is VERY wrong.”\n\nStory about an Christmas e-mail.\n
• #29: “Have you tried turning it off and on?”\n\nThe Big Difference between SiteOps and an Offensive line is treating SiteOps won’t eat you out of house and home.\n
• #30: \n
• #31: There is someone from the Ruby on Rails world who should remain nameless. Let’s call him …“DHH.” This guys is the Jesus of the Ruby World—if Jesus shaved off his beard and starting dressing like a hipster.\n\n\nImage from: http://www.flickr.com/photos/x180/503574487/\nQuote From: http://www.wired.com/wired/archive/14.04/start.html?pg=3\n
• #32: And, according to his talks, he has only two words for me (and you).\n\nThe Gospel According to David.\n
• #33: Now, I’ve been known to swear on occasion . Sometimes I feel you could just replace all my talks with a recording of me saying “fuckity fuck fucking fuck fuck” (coincidentally, this is exactly what it sounds like when I’m conding).\n
• #34: But there is a world of difference between saying this…\n
• #35: And someone saying this to me (and you). This is awful because it ends debate.\n
• #36: And someone saying this to me (and you). It’s bad because it removes all room for debate and discussion. For instance: “Hey, I can’t run Rails in mod_ruby on Apache because of shared memory collisio.”\n
• #37: “Fuck You.” \n
• #38: “Hey, reads and writes on your website growth don’t scale evenly with Moore’s Law so you can’t simply buy a bigger machine to solve your problems.”\n
• #39: “Fuck You.” “Fuck You” is the “Your Momma” of the Rails world.\n
• #40: That isn’t to say I never give a directed fuck.\n
• #41: \n
• #42: That isn’t to say I never give a directed fuck.\n
• #43: That isn’t to say I never give a directed fuck.\n
• #44: I mean the guy should be thanking me. :-)\n
• #45: Even with PHP itself!\n
• #46: …Oh wait, maybe I am.\n\n(It must be true, I read it on the internet.)\n
• #47: Frameworks (like Rails) need to navigate stuck between a Rock and a Hard Place\n\nhttp://sarahdeming.typepad.com/spiralstaircase/2009/10/monsters-of-the-week-scylla-and-charybdis.html\n
• #48: The Rock is Pareto’s principle. It means as your codebase matures, 80% of your time will be spent on 20% of the code… that the Framework doesn’t handle (correctly). In that case you’re better off building a framework that works.\n
• #49: The hard place is maybe something pre-built solves your problem already.\n
• #50: Maybe the problem is already mappable to a forum software and you can download Phorum. Look how much work went into it.\n
• #51: So given the reasoning that Frameworks fail and Ruby on Rails is a framework. The Transitive Property of FAIL means that Rails FAILS. How is this wrong?\n
• #52: It gets back to this Harvard B-school (B stands for bullshit) book.\n\nImage: http://www.flickr.com/photos/tychay/1388234558/\n
• #53: Hard drive manufactures fell to smaller version, mainframes to the miniframe, miniframe to the workstation, workstations to the PC, Sears to WalMart, US Steel vs. small steel mills fail not because of failure to innovate, but because of the appearance of a disruptive technology\n
• #54: In all examples he gives the salient feature of a “disruptive technology” is that it is inferior to the one it replaces. The problem is even in the 1997 book (written during the dotCom boom) he mentioned a lot of examples of disruptive technologies (like Pets.com). Most of them are in the dead pool today.\n
• #55: So you might be thinking… Ahh Ruby on Rails is disruptive. No! That’s not it. Instead, Amazon’s Elastic Compute Cloud: Truly disruptive. On the same days as I got my hands on the PHP terrorist cards I showed on the beginning, this was what I was said during an interview. It is truely unique to be able to see a disruptive technology right from the outset. But cloud computing truly is. It’s inferior and more expensive (to scale) than colocation, but the writing was on the wall even then.\n
• #56: Now compare to IaaS.\n\n1. Starts out more expensive/worse (hell yes!)\n2. Different attributes far from mainstream\n3. Eventually supplants existing system? (OpenStack, OpenCompute -> Nebula announced today)\n
• #57: Now compare to IaaS (and the keynote talk today)\n\n1. Starts out more expensive/worse (hell yes!)\n2. Different attributes far from mainstream\n3. Eventually supplants existing system? (OpenStack, OpenCompute -> Nebula announced today)\n\nNote that shared hosting, managed hosting and colo aren’t going away, they’re shifting to the extremes. We still have supercomputers and compute clouds in a PC/laptop world. Apple still makes the iPod and even a model with a hard drive in it.\n
• #58: Since Rails doesn’t work in shared hosts (remember that joke earlier?), they adopted EC2 (and other things)! And now they are thriving: Capistrano, Ruby version so CruiseControl, GitHub, Heroku… All of it owes it’s life to the disruption of IaaS. And Rails owes its newfound niche to its deficiency in a shared hosting environment.\n
• #59: \n
• #60: \n
• #61: \n
• #62: \n
• #63: Yes, People can do stupid things like try to (re)write Map-Reduce in Rails (without a Reduce function), but that attitude allows them to adopt new things like Git and give us GitHub. The previous examples are Ruby, this is Ruby on Rails.\n
• #64: A big value add in IaaS.\n
• #65: \n
• #66: This is why I choose PHP and why it has become the most popular web-language in the world. It’s our S3.\n
• #67: SIMPLICITY Basically: It’s so simple any moron can learn it.\n
• #68: SCALABLE Basically: Shared nothing means it has nothing to hang yourself with.\n
• #69: STRAIGHTFORWARD Basically: PHP was only designed to solve the web problem. Solve other problems with something else (and link to it in PHP).\n
• #70: This is the dirty little secret of web development: anyone can string together a “SELECT * from users where email=?”, what’s so hard about web development? So if you boil those S3 down to a single S, PHP too succeeds because it sucks. But that suckiness has 40 million developers worldwide!\n
• #71: And now, PHP is ready for it’s own Schylla and Charibdis.\n\nImage from: http://www.flickr.com/photos/davidcoallier/2207776375/\n
• #72: The need for rich ajax driven sites obsoletes traditional advantages fo templating\n
• #73: PHP is an embedded templating language. What is the advantage of a template when everything is in Ajax?\n\nReference: http://antigonemythology.wikispaces.com/file/view/ajax.jpg/33165025/ajax.jpg\n
• #74: Profiling PHP is now a big deal because speed of PHP is becomming the bottleneck\n
• #75: means PHP is no longer waiting on the database. PHP is now the bottleneck. The language need to be performant and developers need to profile. (Note most developers don’t know how to profile code (in PHP it’s with XHProf and kCacheGrind/XDebug) or how to optimize their builds to be performant (replacing Apache with NginX/PHP FPM or using HipHop).\n\nReference: http://web1syndication.com/wp-content/uploads/2010/05/speed.jpg\n
• #76: …except me.\n
• #77: I’ll just chuck in some semicolons.\n
• #78: Why did PHP 6 failed? It failed due to lack of participation. Nobody gave a shit about i18n and all the stuff people wanted (the stuff that helps in performance or building rich Ajax driven websites) was put in PHP 5.2 and 5.3\n
• #79: Let’s talk about the “language needs to be performant part.” We’ll do this with an introductory vignette to part 2. A person at Facebook invented HipHop. Explain how this is different from a JIT (Java, dotNet, JRuby, etc.)\n
• #80: \n
• #81: \n
• #82: This part we discuss the three infrastructures of operations.\n
• #83: This is how PHP became popular. Look at WordPress: 1 click to install on Dreamhost, GoDaddy and the like. (And remember your problem may be mappable to wordpress)\n
• #84: They’re taking away your O-line. Nowadays your ops team looks like this year’s Pittsburgh Steelers. The most difficult part of programming is not in the code, it is in the process. You have to do more with less: We have less testing, so we can release more often. We automate it so we can eliminate user-testing… we had to become more agile for releasing… Now we’ll have to become more agile at hosting. Shared hosting was the start of PHP, but it isn’t it’s end.\n
• #85: Point is automating deployment and testing so you can be agile?\nReleasing often (story of Tagged)\nWhat does that mean?\n
• #86: Most of the discussion isn't in the code, it's in the process. The process is the most ignored part of programming.\n
• #87: This is how a modern high-performance and scalable website (like Facebook) is built.\n
• #88: CPU, RAM, Disk -> Web Server, Memcached, Database.\n\nSomeday you’ll be buying datacenter like you buy a PC (only it’s in the cloud). Or maybe it’s that you’re now buying PC like you buy everything else (Think DropBox or BackBlaze—one of which is built on Amazon Cloud services).\n\nWe consume the same thing already today, they’re called web services. Now simply make the infrastructure a service…\n
• #89: Oh yeah your PC connecting to the internet? That’s your website consuming web services. Remember PCs before the internet? They were islands: Ask yourself why you liked Lorna’s talk.\n
• #90: …this is the cloud computing thing\n
• #91: From “Host Your Web-Site in the Cloud” by Jeff Barr of Amazon\n\nThe big thing is the new model which is pay as you go. It may be more expensive at scale, but you don’t have to anticipate growth with CAPEX.\n
• #92: Before we get to the last part, I want to revisit that “data center is a box” and something that happened recently. Facebook launched the “Facebook Open Compute Project”. What is it?\n
• #93: Announcement: http://www.facebook.com/notes/facebook-engineering/building-efficient-data-centers-with-the-open-compute-project/10150144039563920\n\nCommodity hardware: http://joyeur.com/2011/04/07/facebooks-open-compute-the-dc-is-the-new-server-and-the-rise-of-the-taiwanese-tigers/\n\nhttp://www.readwriteweb.com/cloud/2011/04/what-facebooks-opencompute-mea.php\n
• #94: “On a large scale like this … “opening” something is almost always an effort to commoditize it, leveling the playing field as much as possible and marginalizing competitive advantages that others might have had. [ QUOTE ABOVE ] “Nobody “opens” the parts of their business that make them money, maintain barriers to competitive entry, or otherwise provide significant competitive advantages”\n
• #95: Microsoft and Open PC hardware (eliminated dependency on IBM (and Intel))\nGoogle and applications (browser), platforms (Android), web tech (video) because search business:\nApple and apps\n(I actually don’t look at it as iPhone versus Android. iPhone is a very strong driver of query growth for Google. We also monetize apps through the iPhone…we actually benefit from iPhone’s growth.”)\n
• #96: “But the benefits of commoditization in this area to Facebook are very small…My best guess is that this is primarily for recruiting engineering talent” —Marco Arment.\n\nHe’s wrong with that part. The benefit is to turn Infrastructure into a Commodity. (Dependence on Amazon tools)\n\nRockerfeller and Trains. Impact on people who build large datacenter, Facebook free to move to Amazon services.\n\nWhat is being open sourced? Infrastructure\nOpen Source -> Commoditize your complement\nTherefore: Infrastructure as a Commodity\n\nIf you are a operations person now, then you better get into programming or your next job application will be to a company like Joyent or Amazon.\n
• #97: \n
• #98: \n
• #99: This is what I use. Bought out by RackSpace. There are others\n
• #100: You can go this route but it is a lot of work.\nOn the plus side, since you manage everything you have the least dependencies and are positioned right now to move to wherever is cheapest.\n
• #101: Just don’t make your server look like this.\nOops I offended the one person in the room who uses Gentoo.\nBut look you dont want some person like me using your OS: Look I recompiled everything with CFLAGS=-O9\n
• #102: \n
• #103: \n
• #104: The way to look at the tiers is: you need the base, the services are a nice convenience, and you should probably avoid the management as other competing companies (built on the Base) will do it better/cheaper.\n\nEC2\nS3\nDatabase (RDS, SimpleDB,etc)\n\nAmazon Cloudfrount\nAmazon Route 53\nAmazon Messaging: SQS, SNS\nAmazon Flexible Payment Systems (FPS): their version of PayPal API\nAmazon Mechanical Turk\n\nAWS Management Console\nAWS Elastic Beanstalk - Java.war file creates EC2, etc. instances\nAWS toolkit for Eclipse, Java, PHP etc.\nAWS Virtual Private Cloud (isolated network)\n\n
• #105: This is where programming comes in.\nThere are libraries for this.\n
• #106: \n
• #107: Story of 365Main going down. Story of WP outages\n\nYet, like the famous ocean liner, Amazon's cloud crashed that week, taking with it Reddit, Quora, FourSquare, Hootsuite, parts of the New York Times, ProPublica and about 70 other sites\n
• #108: \n
• #109: Automated deploymnet (many platforms)\n
• #110: This is trying to be the Heroku of the PHP world.\n\nBasically you just point it to your PHP repository and it handles the rest.\nAdd ons\n- Job queue\n- Error logging\n- Couchdb\n- Memcached\n- Mongodb\n\nIt even supports Frameworks like:\nFrameworks:\n- Zend Frameowork\n- Symphony\n- Lithum\n- Solar PHP\n- FRAPI\n\nFramework optimization handled by them is a huge possible performance gain where they can give you more bang for your Amazon buck than you can yourself. Just like Heroku does with Rails.\n
• #111: Frameworks:\n- Zend Frameowork\n- Symphony\n- Lithuum\n- Solar PHP\n- FRAPI\n
• #112: A bit about Frameworks advantages (like Rails)\n
• #113: Vid Luther. Built on AWS. WordPress.\n
• #114: \n
• #115: \n
• #116: Operations requires programming\nProgramming no longer has operations.\nPersonality makes for different peeps\n
• #117: What is the barrier to entry now? vs. when you needed servers and colocation?\n
• #118: Actually this one is true. Security is a low priority and I use services like BackBlaze and DropBox (one basically an Amazon S3 instances)\n
• #119: Built on AWS\n
• #120: Built on AWS\n
• #121: \n
• #122: Performance is important and the performance outside PHP itslef\n
• #123: A Java person says, “Look at it this way, Java is like a knife. A knife can do many things: a good one is essential for cooking, it can cut open packages, cut a cord, and it really comes in handy in a fist-fight. Java can do a lot of things and do them very well.”\nA PHP person, if they had to fight, would rather have something that is designed to solve that one problem, and solve it well.\nA PHP person says, “That knife is nice, but I’d rather have a gun.”\n
• #124: \n
• #125: \n
• #126: It’s on the business track, so after seeing this PHP programmer try to be an system adminstrator, now you see an engineer talk to biz guys.\n\n
• #127: @tychay\ntychay@php.net\nhttp://terrychay.com/\n\nhttp://joind.in/talk/view/3351\n\nPromise to blog more\n
• #128: @tychay\ntychay@php.net\nhttp://terrychay.com/\n\nhttp://joind.in/talk/view/3351\n\nPromise to blog more\n
• #129: @tychay\ntychay@php.net\nhttp://terrychay.com/\n\nhttp://joind.in/talk/view/3351\n\nPromise to blog more\n
• #130: @tychay\ntychay@php.net\nhttp://terrychay.com/\n\nhttp://joind.in/talk/view/3351\n\nPromise to blog more\n
• #131: @tychay\ntychay@php.net\nhttp://terrychay.com/\n\nhttp://joind.in/talk/view/3351\n\nPromise to blog more\n
• #132: @tychay\ntychay@php.net\nhttp://terrychay.com/\n\nhttp://joind.in/talk/view/3351\n\nPromise to blog more\n
• #133: @tychay\ntychay@php.net\nhttp://terrychay.com/\n\nhttp://joind.in/talk/view/3351\n\nPromise to blog more\n
• #134: @tychay\ntychay@php.net\nhttp://terrychay.com/\n\nhttp://joind.in/talk/view/3351\n\nPromise to blog more\n
• #135: So all the speakers at this conference need to send my my cut stat! (That goes double for you, Paul) Triple?\n