Baking Stash in the AWS Cloud at Netflix
0 likes1,792 views
Ed Bukoski, a senior software engineer at Netflix, discusses the implementation and performance of Bitbucket in an AWS cloud infrastructure, highlighting various technical metrics and configurations. The document includes lessons learned from challenges faced during backups and load testing, emphasizing the importance of monitoring and resilient backup scripts. Key takeaways from the experience include embracing cloud technologies, implementing effective metrics, and learning from past mistakes.
Baking Stash in the AWS Cloud at Netflix
- 1. ED BUKOSKI • SENIOR SOFTWARE ENGINEER • NETFLIX • @EBUKOSKI Bitbucket in the AWS Cloud At Netflix
- 2. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
- 3. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
- 4. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
- 5. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
- 6. • Big cool statistic • 2,56 9 • Add-Ons in Marketplace
- 7. BITBUCKET IN AWS LESSONS LEARNED Q&A Agenda
- 8. Pre-Cloud Traditional DC N-Tier Web Applications Relational Databases Cloud Cloud infrastructure - AWS Java, Linux, Apache No SQLHigh Availability Scale Performance Bitbucket At Netflix
- 9. Bitbucket At Netflix Total LOC 1,610,528,526 Total Commits 5,554,379 Builds / Day 7,132 Jenkins Slaves 335 Developers 950
- 10. Bitbucket in AWS
- 11. Cloud Infrastructure Metrics, Monitoring, Backups Load Testing Bitbucket in AWS
- 13. Cloud Infrastructure Amazon Machine Image (AMI) Elastic Compute Cloud (EC2) Instance Elastic Block Storage Volume (EBS) Relational Database (RDS) Elastic Load Balancer (ELB)
- 14. Cloud Infrastructure 1. Stock Atlassian AMI nginx Stash PostgreSQL setups & configs Amazon Machine Image (AMI)
- 15. 2. Custom AMI Cloud Infrastructure Apache/Tomcat How to Make? Organizational Amazon Machine Image (AMI) Metrics Sidecars
- 16. Cloud Infrastructure EC2 Instance c3.8xlarge, 32 vcpu, 60 GB RAM 2x320 SSD ephermal
- 17. Cloud Infrastructure EC2 Instance c3.8xlarge, 32 vcpu, 60 GB RAM 2x320 SSD ephermal EBS Volume General Purpose GP2, 1 TB Stash Home
- 18. Cloud Infrastructure EC2 Instance c3.8xlarge, 32 vcpu, 60 GB RAM 2x320 SSD ephermal EBS Volume General Purpose GP2, 1 TB Stash Home RDS db.m3.xlarge, 4 vcpu, 16 GB RAM 100 GB storage
- 19. Cloud Infrastructure EC2 Instance c3.8xlarge, 32 vcpu, 60 GB RAM 2x320 SSD ephermal EBS Volume General Purpose GP2, 1 TB Stash Home RDS db.m3.xlarge, 4 vcpu, 16 GB RAM 100 GB storage ELB DNS
- 20. Cloud Infrastructure Auto Scaling Group? EC2 Instance c3.8xlarge, 32 vcpu, 60 GB RAM 2x320 SSD ephermal EBS Volume General Purpose GP2, 1 TB Stash Home RDS db.m3.xlarge, 4 vcpu, 16 GB RAM 100 GB storage ELB DNS
- 21. Cloud Infrastructure Bitbucket in AWS Metrics, Monitoring, Backups
- 22. Metrics, Monitoring, Backups 2. Custom AMI Amazon Machine Image (AMI) Metrics Sidecars
- 23. CPU, memory, sessions, JDBC Completely stand-alone WAR Built-in charting Easy, drop in metrics Easy, drop in metrics Completely stand-alone WAR System (CPU, etc), http sessions Built-in charting
- 24. Prana Sidecar “platform” Standalone JVM Ship and index log files Visualize with Kibana Resp times, error rates Logstash,Elasticsearch, Kibana
- 25. Bitbucket DIY Backup Scripts 1. EBS Snapshots 2. Database backups* RDS Database *EBS Instance only custom scripting maint mode — short < 30s Metrics, Monitoring, Backups
- 27. Cloud Infrastructure Metrics, Monitoring, Backups Bitbucket in AWS Load Testing
- 28. Load Testing load testing goals client side measurements 1. Load testing scripts
- 30. 2. Bake an AMI Aminator 3. Create Launch Configs and ASG 4. Scale up ASG to generate load Load Testing
- 31. Scale up ASG
- 32. Server Metrics
- 33. Click here to add page title Client Metrics
- 34. Cloud Infrastructure Metrics, Monitoring, Backups Load Testing Bitbucket in AWS
- 35. Lessons Learned
- 36. Lessons Learned RDS Manual Snapshot Limit max 50 snapshots snapshot error -> kept stash in maint mode
- 37. Lessons Learned Janitor cleans unused AWS infrastructure new rule to clean old RDS snapshots RDS Manual Snapshot Limit backup script more resilient to errors Janitor Monkey
- 38. Volume mount dupe disaster snapshot, attach to new instance two Stashs connected to same database prod test Stash migrated Stash prod database tables mismatched with code prod immediately failed hard Lessons Learned 3.5 context: populate test with prod data test 3.8 connected to prod database started test instance (prod configs!)
- 39. 3.5ish Volume mount dupe disaster prod Lessons Learned 3.5 test 3.8 shutdown test ad-hoc SQL to stabilize database did not restore database from backups analyze liquibase code -> roll-back script upgrade: roll-back then roll-forward
- 40. Bitbucket in AWS - Takeaways Embrace cloud infrastructure Include monitoring and metrics Learn from our mistakes External resources netflix.github.io
- 41. Thank you! ED BUKOSKI • SENIOR SOFTWARE ENGINEER • NETFLIX • @EBUKOSKI