Highly Available and Resilient Multi-Site Deployments Using Spinnaker
1 like534 views
The document outlines strategies for achieving high availability in multi-site deployments using Spinnaker, focusing on application, data, and platform design factors. It discusses concepts such as active-passive and active-active deployments, as well as the importance of managing risk through error budgets and service level indicators. Key takeaways include starting with active-passive architectures, automating processes, and ensuring robust cross-site data replication.
![Stay Connected.
Concourse, and Spinnaker - Today 4:20
Square Pegs, Square Holes: CI/CD That Fits - Yesterday
Scalable, Cloud-Native Data Applications by Example -
Today 2:00 [Geode/Pivotal Cloud Cache]
#springone@s1p](https://image.slidesharecdn.com/highlyavailableandresilient-191017171757/85/Highly-Available-and-Resilient-Multi-Site-Deployments-Using-Spinnaker-45-320.jpg)
![Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Credit
Corey Innis for SRE Illustrations
Customer [0] Team whose work this framework is based on](https://image.slidesharecdn.com/highlyavailableandresilient-191017171757/85/Highly-Available-and-Resilient-Multi-Site-Deployments-Using-Spinnaker-46-320.jpg)
Highly Available and Resilient Multi-Site Deployments Using Spinnaker
- 1. Highly Available and Resilient Multi-Site Deployments Using Spinnaker October 7–10, 2019 Austin Convention Center
- 2. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Crazy Hat Wearers/Senior Platform Architects Koundinya (KD) Srinivasarao Dodd Pfeffer
- 3. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Safe Harbor Statement The following is intended to outline the general direction of Pivotal's offerings. It is intended for information purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking information in this presentation.
- 4. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Agenda ● Rethink How You Achieve High Availability ● Application, Data & Platform Design Factors ● Active-Passive Deployments ● Active-Active Deployments ● Tools / Best Practices / Recommendations
- 5. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ A Framework ● SRE ● Deployment Topologies ○ Active-Passive / Active-Active ● Design Factors: Application / Data / Platform / Other ● Platform/IaaS Prerequisites
- 6. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ “Site Reliability Engineering (SRE) is what you get when you treat operations as if it’s a software problem”1 and aims at developing software systems to solve complex problems. 1 “What is SRE?”: https://landing.google.com/sre/
- 7. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Reliability is a Product Feature Users expect new features We need to respond to users needs Users also expect reliability We need a system that works There is an inherent tension between change and stability We need a way of balancing these needs
- 8. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Failure is Normal Site Reliability Engineering Error Budget (Time) Embrace Risk Eliminate Toil Reliability is Fundamental Service Level Indicator Service Level Objective
- 9. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Balancing Change and Reliability
- 10. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Credit: Corey Innis
- 11. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
- 12. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
- 13. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
- 14. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Managing and Embracing Risk with Error Budgets
- 15. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Your Error Budget is the inverse of your SLO SLO Error Budget 99% < 7 Hours 99.5% < 3.5 Hours 99.9% ~40 Minutes 99.95% ~20 Minutes 99.99% ~4 Minutes 99.999% ~24 Seconds
- 16. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
- 17. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
- 18. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
- 19. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
- 20. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
- 21. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
- 22. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Sources of Risk ● A new release breaks a small set of requests; not detected for a day; quick rollback when detected. ● A configuration mishap reduces capacity; causing overload and dropped requests ● Operator is slow to debug and root cause bug due to noisy alerting ● Other hardware failure; restore from backup required on other site ● Internal Network outage ( between servers ) ● Power failure ● A new release breaks a sizeable subset of requests; unfamiliar rollback procedure extends outage ● Storage overloaded ● Hardware disk failure ● Operator accidentally deletes database; restore from backup is required ● Unnoticed growth in usage triggers overload; service collapses. ● External Internet outage ( to DC )
- 23. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment Topologies
- 24. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Active-Passive Deployment West East
- 25. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Active-Passive Deployment West East The Right Choice for Most Apps Default to this … or at least start from here
- 26. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Active-Active Deployment West East
- 27. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Analysis Tools Active-Passive Worksheet Active-Active Worksheet
- 28. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Design Factors Application Factors What is the impact on the end users? Can users access the app? Can developers successfully cf push? Data Factors What kind of replication do my of apps need? Does this architecture support that? Platform Factors What is the operational experience with this architecture? How are backups and patching affected? What about major upgrades? How is failover handled? What is the ongoing operational cost? Other Factors: Iaas Resources/Capabilities, Network etc...
- 29. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Pet Clinic - Demo Application ● Application Considerations ○ Observability ○ 12 factors: Multiple instances ● Data Considerations ○ Clustered Datastore ○ WAN Replication ● Extended Application Deployment Considerations ○ Separate CI/CD ○ Stateful Inventory ○ Sophisticated Deployment w/ Canary
- 30. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Pet Clinic - Demo Application
- 31. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ PCC Topology 2 Cache Server Cache Server Cache Server Locator 1 1 2 33 1 PCC Client Connection Pool 54.209.251.110 54.209.251.111 54.209.251.112 Locator Cache Server 2 54.209.251.113 Pivotal Platform AZ 1 Pivotal Platform AZ 2 Cache Server 3 1 54.209.251.112 Locator Cache Server 2 54.209.251.113 Pivotal Platform AZ 3 Cluster Spanning Multiple Pivotal Platform AZs PCC Native Redundancy prevent data loss
- 32. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Design Pattern: Foundation Replication Pivotal Platform Pivotal Platform
- 33. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Pivotal Cloud Cache HA and Fault Tolerance Failing data copies are replaced transparently Data is replicated to other clusters and sites (WAN) Split Brain Detection is enabled Client and cluster disconnections are handled gracefully Data is persisted on local disk for ultimate durability “split brain”
- 34. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Continuous Delivery Pipeline Automated builds Unit tests Blue/Green deploys Canary analysis A/B testing Monitoring Security scans Chaos engineering Test-driven dev Iterative coding/fixing Frequent integration Commit code change Store binaries & build artifacts Continuous Integration Continuous Delivery Compliance checks Service tickets Performance tests Security validation
- 35. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Application Centric Control Plan = Turn monitoring into action Deploy safely with speed
- 36. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Automated Canary Analysis Ensures your applications meet your SLAs https://medium.com/netflix-techblog/automated-canary-analysis-at-netflix-with-kayenta-3260bc7acc69
- 37. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Platform/IaaS Prerequisites and Considerations
- 38. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Prerequisites DNS Global Load Balancer ● Configured to point to the two Pivotal Platform foundations ● Each with their own local load balancer ● Global DNS is configured to point to one of the two local load balancers’ VIPs Connectivity/Peering ● Global VNET / Inter-region VPC Peering Identical Release Versions ● Foundations should have the same Pivotal Platform release versions, configurations and capacity, to the extent possible.
- 39. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Active-Active Deployment West East Azure Front Door Global VNET Peering
- 40. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Platform Automation Simplifies Updates & Upgrades ● Works with Concourse and leverages Concourse tasks ● Contains and extends om commands (the Ops Manager API) ● Provides an example pipeline to show how to string together tasks ● Provides a way to extract configuration files, so tasks can support multi-foundation environments. ● IaaS agnostic. Tasks are consistent across IaaS targets. ● Reusable tasks. Run tasks over and over, instead of having to make manual changes. Building blocks for teams to create a repeatable and reusable automated pipeline for upgrading and installing Pivotal Platform foundations.
- 41. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Platform Automation Engine Opinionated Deployment of Concourse, UAA & Credhub Used for Platform Ops Team to run Platform Automation based pipelines
- 42. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Unified Observability
- 43. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Key Takeaways ● Understand and agree on SLO/Error Budget ○ > 99.9% ⇒ Active-Passive or Active Active ○ > 99% ⇒ Single Foundation in a Datacenter/Region ○ Plan for Cluster/AZ capacity to support failed instances from the other Cluster/AZ ● Start with Active-Passive before moving to Active-Active ● Mix of Active-Passive vs Active-Active Based on Workload SLO ● Availability of Data ○ Pivotal data services support cross-site replication (PCC excels at this). Non-Pivotal options exist as well. ● Automate Everything ● Measure Everything, Learn, Adapt
- 44. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ References ● Whitepaper: Multi-Site Pivotal Cloud Foundry Reference Architecture ○ Active-Passive / Active-Active Worksheets ● Petclinic PCC Github Repo ● GitHub Repo: Platform Automation Example ● GitHub Repo: cf-mgmt ● MySQL for Pivotal Platform 2.7, Featuring High Availability
- 45. Stay Connected. Concourse, and Spinnaker - Today 4:20 Square Pegs, Square Holes: CI/CD That Fits - Yesterday Scalable, Cloud-Native Data Applications by Example - Today 2:00 [Geode/Pivotal Cloud Cache] #springone@s1p
- 46. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Credit Corey Innis for SRE Illustrations Customer [0] Team whose work this framework is based on