OSDC 2017 | Something Openshift Kubernetes Containers by Kristian Köhntopp
0 likes29 views
The document discusses the implementation of containerization and Kubernetes at Booking.com, highlighting the shift from monolithic applications to a more agile and scalable infrastructure. It addresses the challenges faced with existing hardware and operational limitations of previous setups, and emphasizes the importance of CI/CD in the development process. Additionally, it outlines the improvements brought by Kubernetes in terms of deployment, networking, and storage management.
OSDC 2017 | Something Openshift Kubernetes Containers by Kristian Köhntopp
- 1. Something Mumble Containers Kubernetes Kristian Köhntopp, Principal in Core.Infra Photo: Kristian Köhntopp The B.boring people
- 2. Containers at Booking Kristian Köhntopp, Principal in Core.Infra Photo: Kristian Köhntopp The B.boring people
- 3. What we do… Photo: booking.com
- 4. A small travel agency in Number of sales ⨉ Transaction value ⨉ Margin The variables structure what you do and how. 4
- 6. Automated baremetal Infra. ServerDB, Nemo - Hands off! Automated box creation, by puppet. 20 min provisioning time (or more). Hardware needs to match workload. No service definitions. Where we are.
- 7. App specific automation on top… Photo: booking.com
- 8. Why this is a problem…
- 9. Current hardware is too powerful One Bladecenter C7000, 16 Blades ~= 896 HT Threads, 3 TB of RAM, 320 GBit/s Network, Can consume 6400 Watt under load. Racks have space for 3-4 of these. Racks have power and cooling for 7000 Watt. Photo: HP
- 10. Core.Infra Be more agile, faster, more flexible. Be able to plan, provide capacity, independent of workload. Machine readable service description, for non-deployment consumption. Be able to scale IT org to 3x current people. Environmental changes…
- 12. So, containers?
- 13. Process PID 18 Program /bin/bash Program /bin/ls Process PID 17 Program /bin/bash fork() exec() exit()wait() contains contains Magic The magic that happens: - Namespaces - Control Groups fork() + fairy dust + exec()
- 14. A single dockerhost… Docker is a completely Dev- centric thing. Not many things Docker does are useful in operations. A single dockerhost can take you very far. 14
- 15. Static linking 2.0 Look, Mom! No RPM!
- 16. Important Corollary: Do CI/CD. You need to own the build chain! CI/CD is not optional. Using a public repo is not helpful (“leftpad”). 16
- 17. Where we come from…
- 18. It came from a Hackathon! Bringing up Docker inside the Apache Mesos/Marathon framework.
- 19. Running stuff. • Blog • Internal websites • ML / bigdata jobs • Java-based web services (many not customer facing) Photo: Kristian Köhntopp
- 20. Results: Works for a small number of workloads. Hard to scale operationally: - No tenant isolation - No integrated service discovery - No blue/green deployment - No auto-scaling - No one-time runs “Like Lego: Works well, but under load parts fall off.”
- 23. Pod Image many cluster nodes Speaking Kubernetes… Sidecar
- 24. “Don’t say container.” POD, image, init-container, sidecar
- 25. Initial Deployments. Iteration 1: - 128 blades (8 bladecenters in 8 racks, 2x 10 GBit/s per blade) Iteration 2: - 128 discretes (8 racks, 1x 25 GBit/s with option to double, potential distrubuted storage playground) Photo: Dell
- 26. A million core computer. “Image based”: Workload can be executed on any free set of resources within a DC. “Unified hardware”: Detach DC design from workload running on top of it. “Location Independence”: Any core in any box, speaking to any disk within the same Data Center. “Strong networking”: Obviously that requires an improved network concept to work.
- 29. L3 Network Leaf and Spine Oversubscription free
- 30. Kris is not a network person.
- 31. You need less SDN than you think
- 32. Cookie cutter topology… Give developers the option to create networks. Do not let them design networks. 32 Common Bus Deployment 1 Deployment 2
- 33. Shared Infrastructure… Logging, Monitoring accessible to all pods. Single hop logging. Requires breakable isolation. 33 Physical Node Pod 1 Pod 2 Common Infra
- 34. No fixed locations, no fixed Image deployed where space allows, with random address. Service: Load balancer with etcd/cluster awareness. 34
- 35. Service Load Balancers for backend discovery. Register Instance Instance Instance
- 36. RequestLookup Service List of Pod Instances mostly static, „watches“.
- 37. RequestLookup Forward to Instance Instance Service In practice a bunch of iptables rules.
- 38. Rescheduling: Instances move around. Instance Rescheduled Instance
- 40. Storage. Cluster Storage: iSCSI from cluster machines to distributed SSD filers, dynamically provisioned. Photo: NetApp
- 42. iSCSI SSD Storage 1. Mount to node 2. bind mount into container Handling persistent storage.
- 43. iSCSI SSD Storage Rescheduled Instance Mount follows Pod iSCSI mount follows Pod instance around.
- 44. 8x Solidfire. Trident for integration. Some growing pains: Latency issues, 64 node limit, … Photo: NetApp
- 45. Things with names… Photo: Kristian Köhntopp
- 46. Stateful Sets Uniqueness guaranteed at any point in time, by ordering teardown, rescheduling. Allows containerization of cluster software. 46
- 47. About to test… Trying to containerize: MySQL BRICK (Elasticsearch) SmartAV 47
- 49. How many IPs per Node? How many per Cluster? IP per container consumes a lot of addresses.
- 50. Legacy Datenbank Gateway Node Gateway Nodes can act as chokepoints.
- 51. Legacy Datenbank Gateway Node ovs-vswitchd Baremetal nodes can be lifted into the Virtual.
- 52. Exit IPs are random, and do not identify a service. Firewall Legacy Server Cluster IP Range ?
- 53. We need to talk about Firewalls… In our case: Rules autogenerated by ServerDB, Puppet class A may speak <port> to B 53
- 54. Partial solutions exist… TLS everything. Use client certs. Handle auth transparently on connect - Trireme. 54 We have not tested any of this, yet.
- 56. What are viable sizes? Photo: Kristian Köhntopp
- 57. Having many clusters: Federation. We have not tested any of this, yet.
- 58. And a few other notes…
- 59. Why not The Cloud™? Photo: Kristian Köhntopp
- 60. Why Containers on Bare Metal? Photo: Kristian Köhntopp
- 61. What about the Monolith?
- 63. ?