A DevOps Journey - CDK Global

Editor's Notes

• #2: -thank InformationWeek, Dr. Dobbs, and CA -the agenda has a title of “Ensuring Predictability and Quality in Unpredictable Business Environments: A DevOps Success Story” I won’t say we’re a success story so I’m going to tell you about our journey
• #3: About CDK -you’ve probably never heard of CDK Global and that’s because up until a couple of months ago, a company by that name didn’t exist -we were part of ADP, who you most likely know as the company that prints your pay checks. We were a division of ADP called Dealer Services, providing technology to the automotive industry -on October 1st, we spun off as an independent, publicly traded company trading on the NASDAQ as CDK -offer technology solutions for consumer research and marketing, dealership visits and sales, negotiation and F&I, and service and parts -ranges from digital advertising to accounting and payroll to telephony systems -if you’ve ever bought a car, chances are you interacted with our systems either from your home while you researched vehicles online or in the dealership when you did your financing About ME -I’ve been in the technology industry for about 14 years -I attended college Marist College in Poughkeepsie, NY where I studied a wide variety of technologies from programming with Java and C++, to virtualization and databases -out of school I worked for a couple of years with a small company as a software engineer working on IP video products -a friend and I decided we should work for ourselves because how hard could it be running a company – it turns out that as much as customers love your products and services, they don’t like to pay you very quickly – lesson learned -one of our biggest customers was a large dealership group, Van Tuyl Auto Group, who was recently in the news since they were acquired by Warren Buffett -this was my segue into the automotive technology space and a few years later, we shut our shop down, sold out to ADP and were part of the Dealer Services family -I bounced around various roles in engineering until landing in my current role about a year and a half back -throughout my tenure, I witnessed many challenges with respect to getting product from an idea into the customers hands and started focusing a lot of effort on solving this so called ‘DevOps’ problem
• #4: -before we get rolling, I’d like to try to level set on a definition DevOps to be used for this presentation -I had my own definition but did some research on how the industry experts were attempting to define it -of course, we all agree with DevOps Borats’ definition [read it] -[read the others] -but ultimately I landed on this [next slide]
• #5: -the key words in here are sharing and common -I spent some time at DevOps days Chicago and many of the talks kept coming back to the essence of DevOps – empathy, not sympathy, empathy -yes, I feel bad for the IT person on pager duty getting woken up 3 times in the night, but I can’t have empathy for him until I’ve been on pager duty myself -then we can start to better understand the problems so we can try to fix them
• #6: -release cycles – major releases were once per year with some patches released during the year – every release was chaos lasting multiple days -DESKING’s OLD RELEASE PROCESS: the deployment process was slow and error prone – one team cherry picked files they knew they changed out of source control, zipped those up, and shipped them since they no source of truth for what should be in production anymore -servers were lovingly handcrafted by artisans, each one with it’s own unique touch -getting those servers provisioned into a working environment took well over a month -the development team “shipped to QA” and it literally meant they sent the code to another team to perform manual testing -R&D reports up to the CTO and IT reports up to the CIO and they had different goals, incentives, and priorities -we have a lot of products on a lot of platforms from .NET, Java, and PHP to Oracle, SQL Server, PostgreSQL, and MySQL -WE ARE A TEXT BOOK DEVOPS problem
• #8: -there is only one answer – start where you are -we booked a room for 2 hours – me, my boss, and the 2 key IT leaders and talked about operational perspectives were important to us – this includes things like an appropriate performance test environments, configuration management, automated deployment and release processes, documented run books and monitoring, disaster recovery, sox compliance and other security perspectives -we then rated our products against each of these perspectives – we were brutally honest – and there was a lot of red!! -we showed it to our CIO and CTO and after their outrage they us the 3 questions from the executive playbook – you know those questions right? How did we get here? What are we doing about it? How do we keep it from happening again? -we picked a select few items to address over the next quarter and then got back together again to see where we stood -each quarter we were improving but the most important thing is that we took control of the situation and started improving while we shared the accountability -which leads me to my first piece of advice
• #10: -so we built on our newly found partnership but found that while we had alignment at the leadership level, the key players who we performing the work were still working in silos -DAMON EDWARDS quote: and there is no greater enemy to throughput and quality than silos. Almost every “devops” problem I’ve seen can be traced back to silos -so what could we do? -we kept thinking, “if only there was a construct that promoted cross-functional teams working towards a shared goal” -at this point, R&D was well on it’s way down the road of Agile but the concepts had not been leveraged in IT so we decided to give it a shot -a cross-functional team meeting daily, tackling action items, and being accountable to each other
• #12: -at this point we were getting better at collaborating, but much of our focus was on IT -as an R&D leader, I knew we needed into improve our flow of business value so I started evangelizing the principles of continuous delivery -LEAD MANAGER STORY-we started small, introducing unit testing, TDD, and continuous builds and for some teams that grew to continuous quality inspection, automated acceptance tests, automated deployments, and more -it’s easy to look at the task as daunting, but we had a team start with a single unit test in an application that was over 75k lines of code, slowly but surely coverage increased while new features were also being created -that team now has over 50% code coverage and is able to release to production every other week – all seeded from 1 passing unit test -DESKING STORY -refer back to cherry picking releases -1 release from June 2012 – June 2013 -3 releases from June 2013 – Dec 2013 -10 releases so far in 2014 -over 2x volume on Black Friday 2014 with no hiccups whereas Black Friday 2014 the system crashed multiple times -all through lots of iterative changes along with a new mindset – a DBA tuning an index here, an engineer optimizing some code there, increased test coverage, etc -so my advice…
• #14: -I heard a great quote at DevOpsDay Chicago, I believe it’s credited to Nathan Harvey, “the only devops tool is the person who calls themselves the director of devops” -But obviously tools have value if wielded correctly -the story I just told about the team that increased code coverage was able to do that using a tool called SonarQube which provides continuous quality inspection – every day they looked at the tool and saw coverage was going up and to the right, motivating them to do more -another team never considered automated deployment until we introduced CA Release Automation to them – after automating their application deployment, they became one of the first groups to adopt LiquiBase for database change management -tools can sometimes be seen as a 4-letter word in the DevOps community, but they play a vital role in it
• #16: -one of the dangers of tools is tool sprawl, and we had it -Jenkins servers under desk, TFS and ClearCase, a little GitHub enterprise here, some Subversion sprinkled there -=