DevConf 2020: Resiliency and availability design patterns for the cloud
Download as PPTX, PDF1 like211 views
The document discusses resiliency and availability design patterns in cloud systems, emphasizing the importance of building resilient software to handle failures effectively. It highlights strategies such as multi-AZ architecture, auto-scaling, and chaos engineering, along with the significant costs associated with application downtime. Overall, it provides insights on leveraging AWS infrastructure to enhance application availability and performance.
DevConf 2020: Resiliency and availability design patterns for the cloud
- 1. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. O N L I N E - D E V C O N F Z A Resiliency and availability design patterns for the cloud Cobus Bernard Senior Developer Advocate AmazonWeb Services 02.04.20 @cobusbernard cobusbernard cobusbernard
- 2. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 3. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Can you guess whatwillhappen?
- 4. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Distributed Systems are hard
- 5. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Failures areagivenand everythingwilleventuallyfail over time. Werner Vogels CTO – Amazon.com “ “
- 6. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Resiliency:Abilityfor asystemtohandle and eventuallyrecover from unexpected conditions conditions
- 7. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Partialfailure mode
- 8. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Why do we build resilient software systems?
- 9. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thecostof downtime Annual Fortune 1000 application downtime costs (IDC) $1.25 to $2.5B Average cost of a data breach (Ponemon Institute) $3.6M Cost/hr of a critical application failure (IDC) $500K to $1M Average cost/hr of downtime (Ponemon Institute) $474K Average cost per lost or stolen record (Ponemon Institute) $141
- 10. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. How do we build resilient software systems?
- 11. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. People Application Network & Data Infrastructure
- 12. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Let’s talk aboutAvailability
- 13. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Systemavailability Availability = Normal Operation Time Total Time MTBF** MTBF** + MTTR* = * Mean Time To Repair (MTTR) **Mean Time Between Failure (MTBF)
- 14. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Reading homework
- 15. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Availabilityinparallel A = 1 – (1 – Ax)2 Part X Part X
- 16. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Availabilityinparallel Component Availability Downtime X 99% (2-nines) 3 days 15 hours Two X in parallel 99.99% (4-nines) 52 minutes Three X in parallel 99.9999% (6-nines) 31 seconds
- 17. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Componentredundancyincreases availability significantly!
- 18. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Global Infrastructure • 22 Regions with 70 Availability Zones • 5 Regions coming soon: Cape Town Jakarta, Milan, Spain, Osaka • 100Gbps redundant network • 99.99% SLA
- 19. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Fully-scaledAvailabilityZone
- 20. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Highlyredundant regional network
- 21. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Region and availabilityzones Region Availability zone a Availability zone b Availability zone c data center data center data center 1 or more data centers per AZ 2 or more AZs per region (new regions min 3) data center data center data center data center data center data center
- 22. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. How about a global architecture?
- 23. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Once upon a time … Origin
- 24. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. And Now … Origin ~300ms
- 25. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Origin Improve latency for end-users Origin
- 26. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Improve availabilityand disasterrecovery Applications in US West Applications in US East Users from San Francisco Users from New York Service 1 Service 2 Service 3 Service 4 Service 1 Service 2 Service 3 Service 4
- 27. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. So should we go for a global architecture?
- 28. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Perfect your regional architecture first!
- 29. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Let’s talk about Multi-AZ
- 30. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi-AZ architecture Region Availability zone a Availability zone b Availability zone c Instances Instances Instances DB Instance DB instance standby Elastic Load Balancing (ELB)
- 31. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi-AZ architecture Region Availability zone a Availability zone b Availability zone c Instances Instances Instances DB Instance DB instance standby Elastic Load Balancing (ELB)
- 32. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi-AZ architecture Region Availability zone a Availability zone b Availability zone c Instances Instances Instances DB Instance DB instance standby Elastic Load Balancing (ELB)
- 33. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi-AZ architecture Region Availability zone a Availability zone b Availability zone c Instances Instances Instances DB Instance DB instance new master Elastic Load Balancing (ELB)
- 34. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi-AZ architecture • Enables fault-tolerant applications • AWS regional services designed to withstand AZ failures • Leveraged by AWS regional services such as Amazon S3, Amazon DynamoDB, Amazon Aurora, Amazon ELBs, etc. Region Availability zone a Availability zone b Availability zone c Instances Instances Instances DB Instance DB instance standby Elastic Load Balancing (ELB)
- 35. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Let’s talk about auto scaling
- 36. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Auto-Scaling FixedVariable
- 37. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Availability zone 1 Auto Scaling group AWS Region Availability zone 2 Auto-scaling for self-healing Elastic Load Balancing (ELB) X
- 38. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Let’s talk about the responsibility models
- 39. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWSoperationalresponsibilitymodels On-Premises Cloud Less More Compute Virtual Machine EC2 Elastic Beanstalk AWS LambdaFargate Databases MySQL MySQL on EC2 RDS MySQL RDS Aurora Aurora Serverless DynamoDB Storage Storage S3 Messaging ESBs Amazon MQ Kinesis SQS / SNS Analytics Hadoop Hadoop on EC2 EMR Elasticsearch Service Athena Firehose
- 40. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Let’s talk about databases
- 41. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Common resiliencyissueswithDatabases?? REPLICATION BACKUPSSCALING
- 42. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 43. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR AZ 1 AZ 2 AZ 3 Network RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR Storage Node Leader PutItem
- 44. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR AZ 1 AZ 2 AZ 3 RR RR RR RR RR RR RR RR RR RR RR RR RR RR RR Storage Node Leader GetItem Network
- 45. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon.com, Nike, Netflix, Duolingo, Lyft, Airbnb, Samsung, Toyota, and Capital One depend on the scale and performance of DynamoDB to support their workloads. 10 trillion requests per day 20 million requests per second
- 46. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Purpose-built databases Relational Key-value Document In-memory Graph Time-series Ledger DynamoDB NeptuneAmazon RDS Aurora CommercialCommunity Timestream QLDBElastiCacheDocumentDB
- 47. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Read-Writeseparation Master Read Replica Read Replica Read Replica Instance InstanceInstance
- 48. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. DatabaseFederation Users DB Products DB Master (Read) Replica Master (Read) Replica Instance InstanceInstance
- 49. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. DatabaseSharding User ShardID 002345 A 002346 B 002347 C 002348 B 002349 A CBA Master (Read) Replica Master (Read) Replica Master (Read) Replica Instance InstanceInstance
- 50. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Let’s talk about backups
- 51. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 52. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Backup service
- 53. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. PreventingAccidentalTableDeletion https://aws.amazon.com/blogs/database/preventing-accidental-table-deletion-in-dynamodb/
- 54. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. PreventingAccidentalTableDeletion (sql)
- 55. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Practice and testrecoveryfrom your backups!!
- 56. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Let’s talk about timeouts, backoff & retries!
- 57. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Users App DB Conn Pool INSERT INSERT INSERT INSERT What happens if the DB “slows down”? Timeout client side Timeout backend side ??
- 58. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. User 1 App DB Conn Pool INSERT Timeout client side = 10s Timeout backend side = default = Infinite Retry INSERT Retry INSERT ERROR: Failed to get connection from pool Retry
- 59. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. https://docs.microsoft.com/en-us/dotnet/api/system.net.httpwebrequest.timeout
- 60. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 61. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-configuration-properties.html
- 62. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. @timeout_decorator.timeout(5, timeout_exception=StopIteration) def timed_get(url): return requests.get(url) https://pypi.org/project/timeout-decorator/
- 63. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. How elsecould wehaveprevented theerror? User 1 DB Conn Pool INSERT Retry INSERT Retry INSERT Retry ERROR: Failed to get connection from pool
- 64. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. User 1 DB Conn Pool INSERT Timeout client side = 10s Timeout backend side = 10s Wait 2s before Retry INSERT INSERT Wait 4s before Retry Wait 8s before Retry Wait 16s before Retry Backing off betweenretries Releasing connectionsBackoff
- 65. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. No jitter With jitter https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ SimpleExponentialBackoffisnotenough:AddJitter
- 66. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Example: add jitter0-1000ms def get_item(self, url, n=1): MAX_TRIES = 12 try: res = requests.get(url) except: if n > MAX_TRIES: return None n += 1 time.sleep((2 ** n) + (random.randint(0, 1000) / 1000.0)) return self.get_item(url, n) else: return res
- 67. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Idempotent operation No additional effect if it is called more than once with the same input parameters.
- 68. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Let’s talk about health checking!
- 69. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Auto Scaling group Service A Availability zone 1 Auto Scaling group AWS Region Service A Availability zone 2 Service BService B database Email Probing for health Cluster
- 70. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Email database Shallowhealthcheck Instance Cache node Cluster Are you healthy? yes
- 71. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. database Shallowhealthcheck Instance Cache node Cluster Are you healthy? yes Email
- 72. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Email database Deep healthcheck Instance Cache node Cluster Are you healthy? yes Are you healthy? yes yes yes yes
- 73. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Email database Deep healthcheck Instance Cache node Cluster Are you healthy? no Are you healthy? no yes yes yes
- 74. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Prioritize shallow health checks during hard times. Cache.
- 75. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Let’s talk about load shedding.
- 76. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 77. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 78. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cheaply reject excess work
- 79. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 80. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Be careful when selecting the right metric
- 81. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Don’tbeoverlyoptimisticandtakeonmorethanyoucan. Findanoperationalmetrictorejectwhatyoucannottakein. Favorcachedandstaticcontent PrioritizeELBhealthcheck(shallow)pings Inanoverloadsituationyouhavepreciousresources,donotletany ofitgotowaste. Load Shedding
- 82. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Let’s talk aboutresiliency (chaos) engineering
- 83. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Fire Drills
- 84. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Chaosengineering https://github.com/Netflix/SimianArmy
- 85. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Chaos Engineeringis the discipline of experimentingon a distributedsystem in orderto buildconfidence in the system’s capabilitytowithstand turbulentconditionsin production.” http://principlesofchaos.org
- 86. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Failureinjection • Start small & build confidence • Application level • Host failure • Resource attacks (CPU, memory, …) • Network attacks (dependencies, latency, …) • Region attacks • “Paul” attack https://www.gremlin.comhttps://github.com/Netflix/SimianArmy https://chaostoolkit.org
- 87. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. STEADY STATE HYPOTHESIS RUN EXPERIMENT VERIFY FIX! PhasesofChaosEngineering
- 88. Thank you! © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cobus Bernard Senior DeveloperAdvocate Amazon Web Services O N L I N E - D E V C O N F Z A @cobusbernard cobusbernard cobusbernard