Creating an open source load balancer for S3
0 likes62 views
This document discusses Safespring's development of an open source load balancer for their object storage solution. It describes Safespring as an infrastructure company that offers storage, compute and backup services from local data centers using open source technologies like OpenStack and Ceph. The document outlines Safespring's goals for a new load balancer to support a hybrid cloud service for customers, and describes the open source components chosen including Bird, Træfik, Etcd, Letsencrypt, Radosgw and Prometheus. It provides details on the load balancer design, configuration management with Ansible, and Safespring's DevOps workflow and tools.
Creating an open source load balancer for S3
- 1. www.safespring.com Anders Bruvik Infrastructure engineer at Safespring Creating an open source load balancer for S3 @bruvik
- 2. Infrastructure company based in Sweden/Norway Offers storage, compute and backup as a service from local data centres Built with Open Source – Openstack/CEHP
- 3. What is SUNET object storage? Private cloud developed by Safespring A solution to data gravity On premises Data stay close to its users Available Expandable Distributed
- 4. The task ● Create a reusable load balancer built from open source components ● Support a hybrid cloud service - we want to deploy in customer datacenter ● Focus on simplicity, reliability, speed We control They control
- 6. The components ● Bird ● Træfik ● Etcd ● Letsencrypt ● Radosgw (Ceph) ● Prometheus
- 7. BGP everywhere Internet as a design pattern for the datacenter cluster network ensure scalability and predictable performance. No surprises now, no surprises in the future. Technically, this means BGP is used everywhere, even for the last hop to each server node.
- 8. ECMP ● We chose ECMP as routing strategy ● Ensures maximum capacity is used ● Achieve redundancy ● Per flow ECMP
- 11. Fast path design
- 12. ● Open Source Load balancer/Reverse proxy written in GO ● Lightweight, fast, easy configurable ● Can dynamically load configuration changes
- 15. Træfik configuration Multiple Træfik instances sharing same configuration Static configuration as files Dynamic configuration stored in etcd cluster
- 16. ETCD ● Distributed key value store ● Used for keeping dynamic Træfik configuration synchronised
- 18. Letsencrypt ● TLS is terminated at the load balancer ● Allows us to use letsencrypt certificates - free and easy ● Træfik supports ACME, data stored in KV- store
- 19. Handling failure Each Træfik instance has multiple backends Using weighting to default to backend on same physical host Monitoring status using systemd service Remove route - restart services
- 20. Monitoring Prometheus is used for monitoring Træfik has a built in metric exporter
- 23. Configuration management We use Ansible for configuration management Heavily investing in Ansible roles to make configuration reusable Separating configuration data and configuration Building tooling to convert Ansible roles to images, docker images or deploy directly
- 24. Safespring DevOps - Workflow What is needed? ● A mechanism to build (Smie - forge). ● A place to store artefacts - could be image, container or binary (Naust - boat house) ● Mechanism for deployment (Seter - settlement) that could describe different runtime environments
- 25. Safespring DevOps - Smie What is Smie? ● Wrapper around Packer (Hashicorp) ● Produces artefacts ● All artefacts can be built separately ● Role: service, endpoint or component
- 26. Safespring DevOps - Naust What is Naust? ● Both source and destination for Smie (cut dependencies to Internet repos) ● Full control over everything built for production ● Protocols: ○ HTTPS/file, S3, Docker Registry ● Protocols depend on target systems ● Everything built get an URI with metadata (type,version, date)
- 27. Safespring DevOps - Seter What is Seter? ● Wrapper around Ansible and Terraform provisioning mechanism ● Describes a system which is a set of artefacts in order to get a component running ● e.g Ceph Object Storage backend needs a set of OSD and RadosGW role images ● Also describes target: physical node, virtual node or container System art 2 Physical art 1 Virtual Contain er art 1
- 28. Safespring DevOps - Advantages ● Update systems faster ○ Lower barrier to changes ● Reproduce systems as needed ○ Build everything with as few dependencies as possible ● Add or change easily ○ Target the affected nodes easily ● Verify that software works as intended ● Scales better with many operators
- 30. Future work Actually deploy this :) Test failure scenarios - chaos engineering Metrics – getting useful Move components to docker images Test and profile different load balance strategies