DevOps Transformations

Editor's Notes

• #2: Hello from Austin, TX! I’m here to share with you a series of four DevOps transformations I’ve been involved in leading in two very different companies. I don’t claim any of these ideas or techniques we used are unique, but I wanted to demonstrate how DevOps concepts can get traction to solve various business problems in a variety of environments.
• #3: I spent the 1990s in Memphis, TN, first working in the giant IT shop at FedEx and then taking on technical leadership at a publishing company growing into the Internet, getting experience in coding, system administration, management, and running large scale Web properties. In 2001 I came to Austin and started in on a series of what would become DevOps transformations in four distinct roles, two at National Instruments and two at Bazaarvoice.
• #4: NI products help scientists and engineers power everything from the CERN supercollider to LEGO robots. Has anyone used LabVIEW?
• #5: My previous company went under in the post-9/11 bust. My passion for Web technology and technical management led to a role back home in Texas at National Instruments. IT at NI was organized around standard enterprise lines. Reporting to the CFO, separate applications and infrastructure organizations. The two primary parts of the applications organization were the groups working on manufacturing and ERP systems based on Oracle E-Business Suite and the Web Team that supported the Web site.
• #7: So keep in mind this was “pre-DevOps” (and even pre-Agile, mostly). We had to dig ourselves out of the hole and then work forward while we were growing and adding new technologies and systems all the time. Like many folks, we were discovering DevOps best practices as we went, by the sweat of our brow.
• #8: In fact some of my fondest achievements from this gig are that the people I had working for me have all gone on to blossom into the areas they wanted to – Peco was our APM COE and now he’s a PM for Riverbed, Josh and James both wanted to get into security and now Josh is head of security for NI and James is notable in the security world, creator of the open source security/CI integration tool gauntlt and working at a security startup. (see http://www.riverbed.com/customer-stories/National-Instruments.html for the source on the APM benefit stats.)
• #9: We got to where we had measurable availability (three nines) and performance goals that we were hitting along with hitting the yearly site growth and sales goals. We had an operational rotation that balanced project work with reactive work for the team. Over 6 years we fixed a lot, started performance and security centers of excellence, had very skilled engineers.
• #10: I knew in my heart something was wrong. We were throwing a lot of smart people at the work we had and had good process, good automation, good collaboration – but we had plateaued. Keep in mind, there hadn’t been the first DevOpsDays yet. Continuous Delivery hadn’t been published. “Dev and Ops Cooperation at Flickr” hadn’t been presented at Velocity yet. We had happened on to a lot of the DevOps approach The dev teams were just starting to use Agile and it was giving us heartache. Discussion of infrastructure test was largely met with blank looks. Infrastructure teams especially were not receptive to a faster cadence. We had gone to the first Velocity conference in 2008 and had been filled with all kinds of new ideas… Also, interaction within Infrastructure was a problem. As our approach and goals began to differ from those of the other teams we experienced conflict – we needed OpsOps, you might say.
• #12: We didn’t know exactly what to do, but we knew that the existing way we worked IT systems would never be fast or agile enough for product purposes. When we got VMWare in IT, it turned a 6 week server procurement process into 4 weeks, only removing the Dell fulfillment time. So we went all in on cloud up front, even though it meant displacing every single tool we were familiar with (none were cloud friendly at the time). Security wise, we got ahead of that by doing security work proactively and being transparent about doing so. The first question every LabVIEW FPGA customer asked was “wait but what about the security of my IP?” And we had a prepared response assuring them “your data is yours” with our CISSP talking about how we do threat modeling and regular security testing and have an incident reporting. And that satisfied every asker!
• #13: We also embarked on writing our own configuration management system, PIE. Why? Simply, chef and puppet existed at the time but didn’t do what we need – Windows support in particular, but also we knew we wanted something that could instantiate and manage an entire system at runtime and understand service dependencies. We figured we’d need to mix Amazon cloud, Azure cloud, NI on premise services, and other SaaS services and a CM system that “puts bits on machines” doesn’t do that. We knew this would be a lot of work that a small team would not be putting towards direct product development and we had to really all look at each other and decide that yes, this was a necessary part of achieving our goal.
• #14: So we went in with a very simple but strict goal of “have that entire phylogical diagram be code, and have it be dynamic as systems change, code is deployed, etc.” PIE had an XML model detailing the systems, the services running on those systems, and their interconnections. It used zookeeper as a runtime registry. Dev, test, and production systems were always identical. Changes were made exclusively through version control. We even used PIE for ad hoc command dispatch so that could be logged. This approach is finally gaining currency especially with docker’s more explicit support for service dependencies, but nothing else did this well for a long time.
• #15: We had some design reviews where the app architect came to me and said “these ops guys are asking questions we just don’t understand… Maybe we should have separate reviews.” I explained that we could learn each other’s language and set up a virtuous cycle of collaboration or continue to split everything and go down the vicious cycle of tickets and process and bottlenecks. So we kept at it.
• #16: Even with building our own automation platform (and other plumbing like login and licensing services), we delivered a product to market each year (plus some other prototypes and internal services), both directly web-facing and integrated with NI’s desktop products. Besides this, our uptime was 100% (vs 98.59% for our online catalog during the same period), we had security we’d never been able to get done (DMZs for example)… It may not be “fair” to compare a greenfield app with our legacy site, but we’d been working on that site for 6 years, and in the end it’s the results that matter. We didn’t have to balance resiliency and innovation, we were getting both with our approach. When we got told “You need to put some of that on Microsoft Azure” it was pretty straightforward to extend PIE to do so. The primary hurdle was more in the sales and marketing of these new solutions – they were coming faster than they were prepared for!
• #17: I think it’s important to note that during this period is when DevOps was coined as a term and took off. I was very, very lucky in that many of the principals happened to come to OpsCamp Austin right after the first DevOpsDays happened and were all abuzz about it. Getting plugged into this wave of sharing was extremely energizing – we kept wondering if we were completely crazy and off track until we found out that so many people were finding the same issues and headed in the same direction.
• #18: Bazaarvoice provides hosted ratings and reviews, along with related services like authenticity, analytics, content syndication between retailers and brands, etc. They’re one of the big Austin tech startup success stories.
• #21: We took a strong stance on dev empowerment and responsibility, and also each team setting their own specific metrics.
• #24: A lot of the existing ops team went to the PRR team. I started with them, got them running on Scrum, worked on oncall, incident command, etc. processes.
• #25: This “legacy stack” is still around now, of course.
• #26: Pretty much it’s all about collaboration, and putting just enough process in place to aid collaboration.
• #29: In the end we had large scale integrated DevOps doing a lot of sustaining work as well as continuing heavy infrastructure development successfully.