Cloud-Scale Kubernetes at eBay
Download as PPTX, PDF4 likes1,242 views
Kubernetes is being used at eBay to manage their large cloud infrastructure. Some key facts: - eBay's cloud infrastructure manages half a million compute cores across 150,000 servers, 200+ PB of data, and handles billions of queries and downloads daily. - They currently use a traditional IaaS-oriented lifecycle for app platforms, but are moving to Kubernetes for its container-based model, declarative abstractions, and centralized state store. - Challenges in adopting Kubernetes include networking (IP routing, density, multi-tenancy), security (IAM integration, container security), storage (integration with block storage), and scale (currently clusters of 5-20k servers each).
Cloud-Scale Kubernetes at eBay
- 2. eBay cloud services ● Cloud Infrastructure and Platform Services ● Data Services ● Database Infrastructure ● Analytics Infrastructure
- 3. Our portfolio Manages ½ Million+ Compute Cores, 150K+ compute servers 200+ PB of Data Infrastructure eBay.com Data and Infra services handles Billions of data Queries per day 20 Billion+ Image Downloads Per Day 300+ TB of logs per day 2 Million Monitoring Metrics Per Second
- 5. app platforms current lifecycle - IaaS oriented
- 6. current Host Runtime - Standardized Deployment Agent
- 7. Kubelet + Container Runtime
- 9. dynamic scheduling with a central cluster manager
- 11. distribute network and other framework functions
- 12. kubernetes Opensource Container based runtime Declarative, application centric abstractions Watchable, updateable central state store Well defined cloud provider and infrastructure plugin model Support from various IaaS clouds (Openstack, public...) Awesome community with great thought leadership Waiting for the evolution of “XPaaS on Kubernetes” marketplace
- 14. challenges
- 15. Challenges: Networking IP Routability ✔ IP Density Multi-tenancy
- 16. overlay based
- 17. L3 routed model
- 18. Plans: Networking BGP to host Layer 7 Load Balancing IPV6 Overlay
- 19. Challenges: Security Integrate with cloud IAM (Openstack Keystone) ✔ Container security (selinux, seccomp, Docker Content Trust) Distributed firewalls Native Public Key Infrastructure on Kubernetes (CFSSL/Lemur) Policy based IAM
- 20. Challenges: Storage Integrate with cloud block storage (Openstack Cinder ) ✔ Local storage leases
- 21. Challenges: Scale Currently has 5-20K servers per AZ Cluster Federation
- 22. Plans: Scale Scale testing lab - work with scale SIG Ubernetes Static provisioning using detached Kubelet
Editor's Notes
- #5: Lots of flexibility now Flexibility comes with do it yourself.
- #7: Very flexible and works great for a wide variety of applications Host runtime is tied to application. Single VM deployment is forced. Application dependencies are not fully captured, assumes host. Framework services are tied to HOST OS capabilities.
- #8: Separate host runtime from application runtime Patching is easier Migration is easier Introduce new application stacks faster Decouple framework services from application stacks Capture application dependencies completely and transactionally
- #9: causes suboptimal allocations
- #10: collect application intents and dependencies schedules workloads efficiently with full cluster state
- #12: distribute network functions application specific functions - sidecars maximally utilize network paths and resources