Autoscaling near-persistent EBS
- 1. Autoscaling near- persistent EBS Emil Filipov AWS Meetup Berlin| November, 2017
- 2. HERE Technologies • The company for location intelligence! • 7000+ employees worldwide • 1000+ employees in Berlin • Navteq (2004), merged into Nokia (2011), split out as HERE (2015) • Privately owned by a consortium of Audi, BMW, Daimler, Intel and others • One of the biggest AWS customers • We’re hiring! https://www.here.com/careers © 2017 HERE | PublicAutoscaling near-persistent EBS | November, 20172
- 3. About me • Software Engineer @HERE Technologies • Certified AWS wrangler • Python zealot • Professional Slavic Pessimist • Working on AWS-centric deployment platforms and automation © 2017 HERE | PublicAutoscaling near-persistent EBS | November, 20173
- 4. 01 History …all good things come out of problems
- 5. The Platform • Dozen identical PuppetMaster EC2 instances in 2 regions • Delivering configuration/changes to several thousand nodes • Each node polling PMs every few minutes • Covering ~100 applications with multiple environments for each • Working across AWS accounts © 2017 HERE | PublicAutoscaling near-persistent EBS | November, 20175
- 6. The Configuration Path • Each app lives in it’s own SCM tree • Each tree sports multiple tags and branches • Each commit to a tree tag/branch is put through a CI system • The artifact out of CI is the state of the tag/branch at that commit • …in the form of an RPM package • PMs install new RPM packages continuously from a central repo © 2017 HERE | PublicAutoscaling near-persistent EBS | November, 20176
- 7. The Problem • For a PM to be functional it needs to install thousands of RPMs • Booting a fresh PM instance takes hours! No dynamic scale Painful updates & maintenance Burning cycles doing the same thing over and over again • Smells Like Datacenter Spirit © 2017 HERE | PublicAutoscaling near-persistent EBS | November, 20177
- 8. 02 Finding a solution …like a pro
- 9. So, I bet you… • Described in exact detail the pain points of the current system • Gathered the requirements a potential solution should fulfill • Conducted scientific research in the space of possible solutions • Compiled reports of the performance characteristics, pros and cons • After a vigorous, but objective discussion, a natural winner emerged Sure… © 2017 HERE | PublicAutoscaling near-persistent EBS | November, 20179
- 10. How about we boot off a backed-up state? • How are we going to do/store backups? EBS Snapshots! • Who’s gonna take those Snapshots? Lambda! • How frequently do we take Snapshots? Every minute! • Who’s gonna make new EBS volumes and attach them to instances? Puppet! (Oh well, let me put a second RPM DB on that EBS volume) © 2017 HERE | PublicAutoscaling near-persistent EBS | November, 201710
- 11. 03 Implementation …turned out to be rather easy
- 12. The DNA © 2017 HERE | PublicAutoscaling near-persistent EBS | November, 201712
- 13. Basic implementation • Lambda function Creates a single Snapshot off the oldest running instance Tag new snapshot with expiration time Delete all expired snapshots (but never if there is only 1 standing) • Configuration management (Puppet) automation Create EBS volume off the freshest Snapshot Wait until volume is available Attach Mount © 2017 HERE | PublicAutoscaling near-persistent EBS | November, 201713
- 14. Achievement unlocked! • Boot times reduced from hours to minutes • Lambda packaged together with the PM CloudFormation stack • Very few settings = simple to add to an existing stack • No penalty for the working instances in IO latency or CPU • Little cost overhead Lambda: less than $0.30/month (once per minute execution) Snapshots: (1 * DataSize + ChangeRate) * S3 price © 2017 HERE | PublicAutoscaling near-persistent EBS | November, 201714
- 15. Here Be Dragons! • EBS is only crash-consistent; fsck is a must • App should ideally be able to handle crashes (but RPM DB does not!) Prepare for corrupted data on disk! • Have to handle EBS leaks separately • Some persistency is lost between the snapshots • One more component to monitor © 2017 HERE | PublicAutoscaling near-persistent EBS | November, 201715
- 16. 04 Now that I have a Hammer …I see a lot of nails
- 17. When is the near-persistent EBS pattern good? You can incrementally sync-up data You have a lot of static data You can afford some data loss • You can do “cheap” sync-up from peers or external location • Small or temporary differences in the dataset are not problematic • Example: Replicated CouchDB, read replicas, resilient distributed storage systems in general • Your application needs to load a lot of data on startup • Data is too big to conveniently use an AMI or download off central storage • Examples: ML corpus, Map Data • Data is volatile and losing a few minutes worth is not problematic • You’re sure of your app’s ability to deal with filesystem crashes • Examples: some metric systems, continuous training (think search term autocomplete) © 2017 HERE | PublicAutoscaling near-persistent EBS | November, 201717
- 18. Comparison with other storage strategies Pattern Near-Persistent EBS EFS/NFS “Golden” EBS Fully persistent - Multi-instance access - Single writer - Linear performance - - AZ independent - © 2017 HERE | PublicAutoscaling near-persistent EBS | November, 201718
- 19. 05 Demo Time
- 20. Source code coming soon at © 2017 HERE | PublicAutoscaling near-persistent EBS | November, 201720 https://github.com/heremaps
- 21. Emil Filipov Email: emil.filipov (here.com) IRC: tie (FreeNode) Thank you!