A brief history of automation in Software Engineering

Editor's Notes

• #5: I left lots of details out and try to focus on stuff not covered by other talks covering quite similar topics ;-)
• #6: I am working on this issue :p
• #8: there is no talk without a wikipedia quote ;-)
• #9: Therefore and also...
• #10: Therefore and also...
• #11: Feedback controllers (which were introduced and adopted in the 1930s in industrial processes were an integral part of defining what is known as the beginnings of automation - the feedback controllers concept is also valid for computer systems)
• #12: Feedback controllers (which were introduced and adopted in the 1930s in industrial processes were an integral part of defining what is known as the beginnings of automation - the feedback controllers concept is also valid for computer systems)
• #13: of course, the unix era also introduced new vendors- though, it often meant vendor lock-in and brittle configuration marathons
• #14: of course, the unix era also introduced new vendors- though, it often meant vendor lock-in and brittle configuration marathons
• #15: (an exception here is cfengine whose first version was developed in 1993 - like it or not) Agile, of course we are here because of this :-) In the 2000s not only “Agile” (Agile manifesto) were introduced into the Software world, but products such as CruiseControl were created In 2004 Sun Microsystems started the Hudson project, which turned 2011 into what we know today as Jenkins (there are other vendors and products, but Jenkins is probably the most notable one) On the operations side tools like Puppet (2005) emerged around the same time (an exception here is cfengine whose first version was developed in 1993 - like it or not)
• #18: Adoption of automated testing and distribution of testing practices. At least one graph with some statical number we need :) The statistic shows the variety of testing tools used in software development worldwide in 2017, based on a survey of development professionals. As of early 2017, 13 percent of respondents said they employed manual testing only.
• #19: it is really both, strategy and culture - peter drucker's famous quote: cutlure eats strategy to breakfast
• #20: automation is highly implicit We often here design thinking but system thinking is quite as important - resp. I guess this we will here in the track
• #30: To reach higher and higher levels of automation machine learning is not only a hype (AI first) but gets more and more important. Machine learning and AI is not only all about automatic decision making and optimization but the idea of the usage of feedback is implicit in most ML/AI applications (e.g. neural networks using what was learned in one layer in the next for example via backpropagation). Example applications are: Source code analysis, anomaly detection, error prediction
• #32: started with an MVP, developed in a hurry - no source control, no automated build process (copy by FTP), totally manual After the first release, new features are added, new developers are hired, more users are on the platform - great! [soon, we will be rich!] After a new release was deployed problems popped up everywhere (bugs already fixed happen again, new features are breaking existing feature and also failing in lots of cases which are not on the “happy path”)
• #33: continue with a makefile, use a VCS (e.g. GIT) the QA process went from manual testing plans to a more automated process when unit tests were added, first executed only on the develops machine, this was already great merging of different branches now is much better, the deployment less error prone - we now had hassle free merges and some basic unit testing Though, merging and testing were still apart from each other - after tests were run on the developer’s machine it was merged and deployed (“worked on my branch and on my machine”).
• #38: The team is very happy on what they have accomplished and they want to finish the day with one more release. All tests were green, but then the site was not usable anymore after deploy - messages about parse errors were found in the logfiles, a little script in one service which didn’t have tests (these index.php entry point style scripts) had a merge conflict and was deployed broken and unnoticed Sanity checks, linting, compile steps were added Flaky unit tests were cleaned up All good?!
• #39: Suddenly, again after some new release, errors when integrating with the external (third party) recommendation system stopped working - now the team added tests for every integration point of third party systems but now, developers were waiting always a long time to finish the tests - the solution was to add to the existing unit and functional test, integration tests which were executed only once a day and not on every run. Yay, we are fail safe and have fast feedback for development! \o/ After the latest release users were complaining about very long response times. After some debugging the found out that some very big images and javascript libraries were loaded synchronously - after cleanup and some fixes they decided to add webpagetest into the build pipeline to test for performance regressions.
• #40: After the latest release users were complaining about very long response times. After some debugging the found out that some very big images and javascript libraries were loaded synchronously - after cleanup and some fixes they decided to add webpagetest into the build pipeline to test for performance regressions. Also, tests were running fine, but after the latest release was deployed, the user was not able to press the checkout button - it was overlayed by a HTML div - the team added Selenium tests to verify the frontend.
• #41: After the latest release users were complaining about very long response times. After some debugging the found out that some very big images and javascript libraries were loaded synchronously - after cleanup and some fixes they decided to add webpagetest into the build pipeline to test for performance regressions. Also, tests were running fine, but after the latest release was deployed, the user was not able to press the checkout button - it was overlayed by a HTML div - the team added Selenium tests to verify the frontend.
• #42: After the new killer feature for google docs integration was released, the marketing team launched some huge campaigns on Twitter and Facebook. The site crashed abruptly (the database was overloaded with queries to the database backend which were missing an index but also could have been cashed partially). Besides regular performance testing on their live environment before special events they added the performance plugin to their Jenkins setup and now are running before every deploy a load test against their staging environment. Plan: regular performance testing on their live environment And: added the performance plugin to their Jenkins setup and now are running Jmeter tests before every deploy in staging environment
• #43: Performance and functionality is now [almost always] guaranteed to be great on each deploy, nothing can happen anymore. Suddenly the Head of Security arrives and tells that some hackers were able to get access to other user accounts by a XSS attack and also deleted part of the address database via SQL injection. After they hired an external pentesting team to do a thorough security audit and fixed all security issues, the team decided to also add a Security testing step into their deploy pipeline and used w3af with a custom rule set while also put on their roadmap the evaluation of non-free products (because security is that important).
• #44: Even when unit tests were added all over the code base, you need to maintain them to get real value out of it. Treat them as first class citizens in your code base (clean code, refactoring, etc.). Use mock data which make sense, otherwise soon tests waere failing randomly on each beginning of a month (flaky tests) and it was important to clean up. Sometimes problematic is to use open source software which doesn’t have a public test suite (this was the case with magento at Dafiti) Also, tests were running fine, but after the latest release was deployed, the user was not able to press the checkout button - it was overlayed by a HTML div - the team added Selenium tests to verify the frontend.
• #45: To get timely feedback the team found various ways to integrate notifications into their build pipeline and are now getting informed via Chat (slack channel) and email when something went wrong.
• #46: Pretty great job! The releases were much better and bug free and everyone happy. But there was still a lot of pain when things went live and it was discovered that the CI server used other libraries than the live environment.
• #47: The team made some research on Environment provisioning software and compared between Puppet, Chef, Ansible and cfengine. Because a new hire in the operations team (now called SRE) already had some experience with Puppet they settled on Puppet. The complete setup of the environment is available in configuration files and scripts and can be recreated in a consistent manner everytime needed.
• #51: Even though the application were behaving very well in lots of cases the Operations (or fancy wise SRE) had often headache when it came to spikes in site usage, such as marketing campaigns. Beside this they got pressure to lower costs for infrastructure - to solve those contradictory problems they enabled autoscaling in their AWS setup (this works as well for GcP or Azure). Being more cloud agnostic and following the latest trends of containerization (e.g. Docker) they are also investigating the usage of Kubernetes.
• #52: The team very proud of their test, build, deployment and infrastructure step wants to go a step further. After they read some papers and heard of trends in machine learning they added anomaly detection for their live stack and software error prediction into their code analyzer which is again part of the build pipeline. They also have a bunch of other quite interesting things to evaluate such as chaos engineering where they want to use Netflix’s Chaos monkey to randomly shut down parts of their infrastructure and the automation of autoscaling, error prediction and monitoring should keep them safe. Canary releases and blue green deployments are also there to get tested.