Characterizing Defective Configuration Scripts Used for Continuous Deployment
Download as PPTX, PDF0 likes135 views
The document explores the characterization of defects in infrastructure as code (IaC) scripts to assist software practitioners in prioritizing their validation and verification efforts. It identifies key operations associated with defective IaC scripts, such as filesystem operations and user account management, and discusses methods for building predictive models using text features. The findings highlight the prevalence of defects across various datasets from Mozilla, OpenStack, and Wikimedia, stressing the importance of awareness and targeted efforts in addressing these issues.
Characterizing Defective Configuration Scripts Used for Continuous Deployment
- 1. Characterizing Defective Configuration Scripts Used for Continuous Deployment Akond Rahman and Laurie Williams aarahman@ncsu.edu https://akondrahman.github.io/
- 2. Outline • Motivation • Goal • Research Questions • Methodology • Dataset • Empirical Findings • Summary 2
- 3. Motivation: Continuous Deployment 3 Continuous Integration Integrate software changes regularly Continuous Deployment Integrate software changes regularly Automatically deploy software changes rapidly
- 4. Motivation: Continuous Deployment https://en.facebookbrand.com/assets/f-logo https://www.stitcher.com/podcast/etsy https://brand.netflix.com/en/assets/ Facebook Etsy Netflix 4
- 5. Motivation: Continuous Deployment Summit Continuous Deployment Summit (2015, 2016, 2017) Dr. Chris Parnin Dr. Laurie Williams 5
- 6. Motivation: Infrastructure as Code ( IaC) Continuous Deployment Summit (2015, 2016, 2017) 6 Ansible Chef Puppet
- 7. Motivation: Defects in IaC scripts • Defects in IaC scripts can have serious consequences 7https://wikitech.wikimedia.org/wiki/Incident_documentation/20170118-Labs Wikimedia Commons
- 8. Goal To help software practitioners to prioritize validation and verification efforts for infrastructure as code (IaC) scripts by identifying the operations that characterize defective IaC scripts 8
- 9. Research Questions • RQ-1: Which are the operations that characterize defective infrastructure as code (IaC) scripts? How frequently do the identified operations appear in IaC scripts? • RQ-2: How can we build prediction models for defective infrastructure as code scripts using text features? 9
- 10. Background case $::operatingsystem { ‘CentOS’: { exec { "install iptables": command => "/bin/yum install iptables -y", creates => "/sbin/iptables"; } } default: { fail("cannot install on $::operatingsystem") } } https://puppet.com/company/press-room/media-kit 10
- 11. Methodology • Dataset Construction • Empirical Analysis 11
- 12. Methodology: Dataset Construction Extract Commit IaC Repositories 12 Rater-1 Rater-2 Disagree? Mark as defect or non-defect Resolver Mark as defect or non-defect No Yes
- 13. Methodology: Empirical Analysis Dataset Feature Extraction Correlating Features Answer to RQ1 Answer to RQ2 Bag of words, TF-IDF Principal Component Analysis Qualitative Analysis Random Forest, Parameter Tuning, AUC, F1 13
- 14. Dataset: Sources Mozilla Openstack Wikimedia Commons https://blog.mozilla.org/opendesign/arrival/ https://www.openstack.org/brand/openstack-logo/ https://commons.wikimedia.org/wiki/Category:Logos 14
- 15. Dataset Mozilla Openstack Wikimedia Time Period Aug, 2011- Sep, 2016 Mar, 2011- Sep, 2016 Apr, 2005 – Sep, 2016 IaC Code Size (LOC) 30,272 122,083 17,439 Defect-related Commits 558 of 3074, 18.1% 1987 of 7808, 25.4% 298 of 972, 30.6% Defective IaC Scripts 259 of 580 (44.6%) 810 of 1383 (58.5%) 161 of 296 (54.4%) 15
- 16. Answer to RQ1: Which are the operations that characterize of defective infrastructure as code (IaC) scripts? • Filesystem operations • Infrastructure provisioning – Build systems – Data analytics systems – Database systems – Web server systems • Managing user accounts 16
- 17. Answer to RQ1: Filesystem operations 17
- 18. Answer to RQ1: Infrastructure provisioning • Build systems 18
- 19. Answer to RQ1: Infrastructure provisioning • Data analytics systems 19
- 20. Answer to RQ1: Infrastructure provisioning • Database systems 20
- 21. Answer to RQ1: Infrastructure provisioning • Web server systems 21
- 22. Answer to RQ1: Managing user accounts 22
- 23. Answer to RQ1: Frequency 21.7 14.5 23.4 6.9 18.9 17.9 2.5 1.1 1.6 0 5 10 15 20 25 Mozilla Openstack Wikimedia PercentageofIaCScripts Filesystem Infrastructure provision User account 23
- 24. Answer to RQ2: How can we build prediction models for defective infrastructure as code scripts using text features? 0.76 0.59 0.68 0.75 0.55 0.56 0 0.2 0.4 0.6 0.8 Mozilla Openstack Wikimedia MedianAUC Bag-of-words TF-IDF 0.74 0.71 0.73 0.72 0.74 0.7 0.65 0.7 0.75 Mozilla Openstack Wikimedia MedianF-Measure Bag-of-words TF-IDF 24
- 25. Limitations • Dataset construction • Text mining 25
- 26. Implications • Raise awareness amongst practitioners when doing certain operations • Prioritizing verification and validation efforts based on identified operations 26
- 27. Summary Answer to RQ1: Frequency 21.7 14.5 23.4 6.9 18.9 17.9 2.5 1.1 1.6 0 5 10 15 20 25 Mozilla Openstack Wikimedia PercentageofIaCScripts Filesystem Infrastructure provision User account aarahman@ncsu.edu akondrahman https://akondrahman.github.io/ Contact Answer to RQ2: How can we build prediction models for defective infrastructure as code scripts using text features? 0.76 0.59 0.68 0.75 0.55 0.56 0 0.2 0.4 0.6 0.8 Mozilla Openstack Wikimedia MedianAUC Bag-of-words TF-IDF 0.74 0.71 0.73 0.72 0.74 0.7 0.65 0.7 0.75 Mozilla Openstack Wikimedia MedianF-Measure Bag-of-words TF-IDF 27 Motivation: Defects in IaC scripts • Defects in IaC scripts can have serious consequences 6https://wikitech.wikimedia.org/wiki/Incident_documentation/20170118-Labs Wikimedia Commons