Continuous delivery and the challenges for the test automation
- 1. CONTINUOUS DELIVERY AND THE CHALLENGES FOR THE TEST AUTOMATION DISA KAKKO (MAY 2016) DISA.KAKKO@BASWARE.COM HTTPS://WWW.LINKEDIN.COM/IN/DISAKAKKO
- 3. CONTINUOUS DELIVERY- IDEAL SITUATION • Continuous integration against continuous test automation. • Continuous automatic builds and release packages. • Automatic deployment to the QA and production environments (includes automatic verification of the deployment successfulness). • QA personnel is focused on testing the new functionality with exploratory testing. Test automation covers smoke and regression testing of the existing functionality on all test automation levels. • Monitoring data from the production environment is gathered and the information is utilized to develop each phase preceding the production deployment.
- 4. REALITY? • Some of the teams in the same SW program/project are doing test automation for all levels of sw (unit/API/system), but not all teams. In some cases some test automation level is skipped completely (e.g. unit tests). • There is no pressure to do test automation since the QA personnel will manually test all new features and the existing functionality for every code change. • DoD does not exist or it is not followed. • Release packaging and deployment are done manually. Deployment verification is done by performing manual tests. • Monitoring of the production environment does not exist or the monitoring data is not utilized by the R&D.
- 5. WHAT KIND OF TEST AUTOMATION COVERAGE CONTINUOUS DELIVERY REQUIRES
- 6. CHALLENGES IN THE SOFTWARE DEVELOPMENT- CONSEQUENCES TO THE TEST AUTOMATION?
- 7. WHAT CAUSES THE AVALANCHE? • ”We’re too busy” is often the excuse for dismissing the needed internal QA actions for R&D (refactoring, test automation, static analysis, etc). It is more important to get features released, than to do them properly the first time (which would mean test automation and non-functional aspects taken into account for the new functionality). • QA actions for the R&D (such as test automation or refactoring) is put to the same backlog with the new features. This way the responsibility of doing quality is outsourced to the product management. • If there is a pressure from the sales promises for the new features, the internal QA actions for the R&D (such as test automation work) is moved down in the backlog. Eventually test automation does not get done. -> test automation is not seen as part of the new feature development.
- 8. WHAT CAUSES THE AVALANCHE? • There is no advocate for the test automation work within a team (e.g. the lead developer). • A viewpoint that the QA personnel will take care of the quality (and therefore all leaked bugs are seen as the QA personnel’s fault, not as the team’s fault). • Test automation framework that would suite the team’s needs is missing. • No-one is following the DoD.
- 9. WHAT CAN BE DONE WHEN THE TEST AUTOMATION COVERAGE IS POOR?
- 10. HOW TO CHANGE THE ORGANIZATION CULTURE TO BE ”PRO TEST AUTOMATION” • Velocity allocation within each team for team internal QA development work -> Effort estimates for the new features should include test automation work. • A test automation advocate for each team. • TDD/BDD • Test automation creation based on the leaked bugs. • Top-down approach when starting to create the missing test automation coverage (unit test level is a ”lost case” when there hasn’t been any test automation done). • Production monitoring data utilization (e.g. estimates how data
- 11. HOW TO CHANGE THE ORGANIZATION CULTURE TO BE ”PRO TEST AUTOMATION” • Test automation framework taken into use that addresses the needs of the team. • Automatic tests as part of the CI system and the radiator. • Team’s commitment to the test automation work: • Responsibility agreements (developers focusing on the unit/integration test automation and QA personnel to the UI/system test automation). • Test follow-up responsibilities within the team (test status from the CI and updating the tests when needed). • DoD dicipline (minimum set that the team will commit to). • Test automation work done as part of the feature branch work (or behind the same feature flag)
- 12. CASE: BASWARE – ACTIONS TAKEN • Started to automate the different phases of the deployment and the checking of the deployment successfulness. • Setting up a system for generating the test environments automatically. • Automatic deployment of builds into the automatically generated environments. • System level smoke and regression test automation. Also all APIs covered with test automation. • Test data tool development based on the production monitoring data (possibility to generate average data amounts or amounts greater than the known maximum amounts). • Rotating test automation upgrade responsibility within each team (to avoid test automation to be commented out instead of fixing the outdated tests).
- 13. THANK YOU! • Thoughts that came to Your mind based on this speech? • Questions?