EuroPython 2014 Devops Risk Mitigation
2 likes3,402 views
The document discusses the importance of risk mitigation in DevOps practices, emphasizing a collaborative environment where development and operations teams learn from each other. It outlines testing methodologies including unit and system tests, alongside the need for continuous deployment and automation in creating a reliable infrastructure. The document also highlights the significance of using infrastructure as code and monitoring to enhance deployment processes and reduce downtime.
EuroPython 2014 Devops Risk Mitigation
- 1. DevOps Risk Mitigation www.immobilienscout24.de Berlin | 23.07.2014 | Schlomo Schapiro Systems Architect, Open Source Evangelist License: http://creativecommons.org/licenses/by-nc-nd/3.0/ Test Driven Infrastructure @schlomoschapiro
- 2. Expensive Fix Cheap Fix Costs Of Finding Bugs In Production
- 4. Plan Design Budget Develop Test Plan Design Budget Develop Test RUN Proxy config Database borked „Buy Now“ button broken MTA drop all mail Load Balancer Configuration Everything costs only 0 € Login possible without password NFS not available DB Replication stopped No ads shown Broken CSS / JS Tomcat won't start/stop Service user not defined sudoers invalid
- 5. DevOps: Respect & Learning DEV learn from OPS to think about: ● Resources (CPU, RAM, Disk) ● Services (Start, Stop, Status) ● Dependencies (Start DB before App) ● Logfiles (Rotate, Remove) ● Disk Space ● Monitoring and Alarming ● ... OPS learn from DEV to think about: ● Incremental Improvement ● Infrastructure as Code ● Version Control System ● Coding (OO, Functions, Libraries …) ● Code Quality ● Unit & Integration Tests ● Test Automation ● ...
- 7. Unit Tests Test the smallest possible components in an artifical environment. System Tests Test the entire application in a real(istic) environment together with other applications.
- 8. Part of build process Syntax checks Scripts Config Files Data Files Unit tests for functions/libs Run program with test data Check result Check program behaviour with wrong/broken test data Also run on Developer desktop Quick feedback (~ seconds) Install on test server Run tests from outside HTTP calls Send emails Try to login Run tests from inside Remote Exec (rsh, ssh …) http://go.schapiro.org/rshpitfall Service Start, Stop & Status Modify server to create good & bad test scenarios Reboot Unit Tests System Tests
- 9. Unit Tests
- 10. %prep %setup -q %install install … %{buildroot}/… install … %{buildroot}/… %files %defattr(-,root,root,-) /...
- 11. BuildRequires: sudo %build set -e visudo -c -f sudoers %install install -m 0440 sudoers -D %{buildroot}/etc/suoders.d/%{name} %files%defattr(-,root,root,-) /etc/suoders.d/%{name}
- 12. BuildRequires: PyYAML, pylint %build set -e # syntax checks bash -n my_script.sh # Should be valid python code pylint -E yum-repo-propagate # should be valid YAML file python -c "↩ import yaml↩ yaml.safe_load(open('config.yaml'))↩ "
- 14. Build Auto- mation Source Code (SVN) Monitor Changes Build Server Run Build Job ● Check out source ● Run Unit Tests ● Create RPM ● Source Code DEV YUM Repo Upload PRO YUM Repo Propagate RPM Test Server Deploy and Run TestJob yum Prod Server Deploy to PROD yum
- 15. Build Auto- mation Source Code (SVN) Monitor Changes Build Server Run Build Job ● Check out source ● Run Unit Tests ● Create RPM ● Source Code DEV YUM Repo Upload PRO YUM Repo Propagate RPM Test Server Deploy and Run TestJob yum Prod Server Deploy to PROD yum
- 16. SAN mount service Test via rsh Mock SAN devices with losetup service start, stop mounts/umounts Error handling
- 17. HTTP Proxy Configuration X-Forwarded-For header spoofs source Check result for ERR_ACCESS_DENIED Run tests for all function groups! Test Server GET http://external.com/ X-Forwarded-For: 10.11.12.01 502 Bad Gateway✔ GET http://external.com/ X-Forwarded-For: 10.34.56.01 403 Forbidden ERR_ACCESS_DENIED by proxy server ✘ ✘
- 18. VM Provisioning & Kickstart Installation Test via HTTP API Create broken VMs and check error reports Create valid VM and install Linux OS Scrape VM screen via OCR http://github.com/Immobilienscout24/lab-manager-light
- 19. 34 35 36 2 3 53 87 88 89 90 91 92 93 94 95 96 97 TIME Continous Live Deployment Deploy every application when it is ready. Automate the delivery chain from source till production.
- 20. Low Risk – Lots of Fun http://go.schapiro.org/slides
- 21. Kontakt: Immobilien Scout GmbH Andreasstraße 10 10243 Berlin Fon: +49 30 243 01-1229 Email: schlomo.schapiro@immobilienscout24.de URL: www.immobilienscout24.de Thank you very much! Please contact me for further questions and discussions.