DevOps Patterns & Antipatterns for Continuous Software Updates @ DevOps.com April 2020
1 like1,133 views
The document discusses DevOps patterns and anti-patterns for continuous software updates, emphasizing the importance of timely identification, fixing, and deployment in the software development lifecycle. It highlights various strategies for effective updates, such as automated deployment, progressive delivery, rollback capabilities, and zero downtime updates. The ultimate goal is to adopt a 'liquid software' vision that allows for frequent, seamless updates rather than infrequent, large-scale updates.
DevOps Patterns & Antipatterns for Continuous Software Updates @ DevOps.com April 2020
- 1. DevOps Patterns & Antipatterns for Continuous Software Updates “What can possibly go wrong?!”
- 3. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 6. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 7. “As everycompanybecome asoftware company, Security vulnerabilities are the new oil spills” @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 9. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes Identify Fix Deploy
- 12. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 13. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 14. Identify As fast as possible Fix As fast as possible Deploy As fast as possible @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 18. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 19. This is not a new idea! XP: short feedback Scrum: reducing cycle time to absolute minimum TPS: Decide as late as possible and Deliver as fast as possible Kanban: Incremental change @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 21. @jbaruch #dockercon jfrog.com/shownotes@ErinMeyerINSEAD’s “Culture Map” 🎩
- 22. shownotes http://jfrog.com/shownotes Slides Video Links Comments, Ratings Raffle @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 24. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 25. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 26. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 29. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 30. Do we want it? Let’s update! Are there any high risks? How about no Do we trust the update? Update available No Yes Yes Yes NoNo
- 31. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 32. number of artifacts as a symptom of complexity @jbaruch @jfrog #LiquidSoftware www.liquidsoftware.com IoT Serverless Docker Infrastructure as Code Continuous Delivery Continuous Integration Agile Microservices 2000 Today
- 33. The problem is not the code, it’s the data. Big data. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 35. Do we want it? Let’s update! Are there any high risks? Can we verify the update? How about no Do we trust the update? Update available No Yes Yes Yes Yes NoNoNo Time consuming verification
- 37. Features that we want Acceptance tests costs @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 39. Do we want it? Let’s update! Are there any high risks? Update available No Yes No one asked you (auto update) Your browser Twitter in your browser Twitter on your smartphone Your smartphone OS?!
- 40. What can possibly go wrong?
- 41. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 43. Continuous updates pattern: Local rollback Problem: update went catastrophically wrong and an over the-air patch can’t reach the device Solution: Have a previous version saved on the device prior to update. Rollback in case problem occurred @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 44. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 48. Continuous updates pattern: OTA software updates Problem: physical recalls are costly. Extremely costly. Also, you can’t force an upgrade. Solution: Implement over the air software updates, preferably, continuous updates. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 49. continuous OTA updates are like normal OTA updates, but better @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 52. Continuous updates pattern: continuous updates Problem: In batch updates important features wait for non-important features. Solution: Implement continuous updates. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 53. You thought your problems are hard? Things under your control Server-side Updates IoT (Mobile, Automotive, Edge) Updates The availability of the target ✓ ✕ The state of the target ✓ ✕ The version on the target ✓ ✕ The access to the target ✓ ✕ @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 55. KNIGHT-MARE New system reused old APIs 1 out of 8 servers was not updated New clients sent requests to machine contained old code Engineers undeployed working code from updated servers, increasing the load on the not-updated server No monitoring, no alerting, no debugging @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 56. Continuous updates pattern: Automated deployment Problem: People suck at repetitive tasks. Solution: Automate everything. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 57. Continuous updates pattern: frequent updates Problem: Seldom deployments generate anxiety and stress, leading to errors. Solution: Update frequently to develop skill and habit. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 58. Continuous updates pattern: state awareness Problem: Target state can affect the update process and the behavior of the system after the update. Solution: Know and consider target state when updating. Reverting might require revering the state. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 60. Cloud-dark New rules are deployed frequently to battle attacks Deployment of a single misconfigured rule Included regex to spike CPU to 100% “Affected region: Earth” @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 61. Continuous updates pattern: Progressive Delivery Problem: Releasing a bug affects ALL the users. Solution: Release to a small number of users first effectively reducing the blast radius and observe. If a problem occurs, stop the release, revert or update the affected users. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 62. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 63. Continuous updates pattern: observability Problem: Some problems are hard to trace relying on user feedback only Solution: Implement tracing, monitoring and logging @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 64. Continuous updates pattern: Rollbacks Problem: Fixes might take time, users suffer in a meanwhile Solution: Implement rollback, the ability to deploy a previous version without delay @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 65. Continuous updates pattern: feature flags Problem: Rollbacks are not always supported by the deployment target platform Solution: Embed 2 versions of the features in the app itself and trigger them with API calls @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 67. Continuous updates pattern: zero downtime updates Problem: You will probably loose all your users if you shut down for 5 weeks to perform an update. Solution: Perform zero- downtime OTA small and fequent continuous updates. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 68. Continuous updates Frequent Automatic Tested Progressively delivered State-aware Observability *Local Rollbacks @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 69. Do we want it? Let’s update! Are there any high risks? Do we trust the update? Update available No Yes Yes Yes Sure, why not? (auto update)
- 70. Our goal is to transition from bulk and rare software updates to extremely tiny and extremely frequent software updates; so tiny and so frequent that they provide an illusion of software flowing from development to the update target. We call it the Liquid Software vision. ” @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 71. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 72. Corner cases? @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 75. @jbaruch #LiquidSoftware @devopsdotcom http://jfrog.com/shownotes
- 76. Q&A and twitter ads @jbaruch #LiquidSoftware #DevNexus https://liquidsoftware.com https://jfrog.com/shownotes