Container Networking Meetup March 31 2016
Download as PPTX, PDF2 likes1,102 views
The document discusses Project Calico, highlighting its role in networking for containerized environments and the challenges faced in achieving efficient network configuration. It outlines key issues such as scaling, IP address limitations, and evolving orchestration platform support, while advocating for a simpler, more effective routing approach. Additionally, it emphasizes the need for proper configuration to avoid sub-optimal performance and discusses the implications of container networking on cloud-based infrastructures.
Container Networking Meetup March 31 2016
- 1. Project Calico is sponsored by@projectcalico Sponsored by Networking in a Containerized Data Center: the Gotchas! MICROSERVICES FOR ENTERPRISES MEETUP Andy Randall | @andrew_randall Palo Alto, March 31, 2016
- 2. Project Calico is sponsored by@projectcalico (n) North American. “an instance of publicly tricking someone or exposing them to ridicule, especially by means of an elaborate deception.”
- 3. Project Calico is sponsored by@projectcalico
- 4. Project Calico is sponsored by@projectcalico Run anywhere Simple Lightweight Standard Speed Cloud Efficient
- 5. Project Calico is sponsored by@projectcalico
- 6. Project Calico is sponsored by@projectcalico The original “container approach” to networking All containers on a machine share the same IP address Gotcha #1: WWW1 WWW2 80 80 Proxy 8080 8081 Still most container deployments use this method!
- 7. Project Calico is sponsored by@projectcalico World is moving to “IP per container” Container Network Interface (CNI) Container Network Model (libnetwork, 0.19) net-modules (0.26) (future: CNI?)
- 8. Project Calico is sponsored by@projectcalico We’ve solved “IP per VM” before… VM 1 VM 2 VM 3 Virtual Switch
- 9. Project Calico is sponsored by@projectcalico We’ve solved “IP per VM” before… VM 1 VM 2 VM 3 Virtual Switch VM 1 VM 2 VM 3 Virtual Switch
- 10. Project Calico is sponsored by@projectcalico Consequences for containers (gotcha #2): Scale Hundreds of servers, low churn Millions of containers, high churn
- 11. Project Calico is sponsored by@projectcalico pHost 1 Virtual Switch / encapsulation vNIC pNIC vNIC VM1 Consequences for containers (gotcha #3): Layering Packets are double encap’d! Container A Container B Container C Virtual Switch / encapsulation veth0 veth1 veth2 pHost 2 Virtual Switch / encapsulation VM2 Container D Container E Container F Virtual Switch / encapsulation pNIC vNIC vNIC veth0 veth1 veth2 Physical Switch
- 12. Project Calico is sponsored by@projectcalico Consequences for containers (gotcha #4): walled gardens Legacy App pHost 1 Virtual Switch / encapsulation vNIC pNIC vNIC VM1 Container A Container B Container C Virtual Switch / encapsulation veth0 veth1 veth2 Physical Switch
- 13. Project Calico is sponsored by@projectcalico “Any intelligent fool can make things bigger, more complex… It takes a touch of genius – and a lot of courage – to move in the opposite direction.”
- 14. Project Calico is sponsored by@projectcalico A Saner Approach: just route IP from the container pHost 1 Virtual underlay vNIC pNIC vNIC VM1 Container A Container B Container C Linux kernel routing (no encapsulation) veth0 veth1 veth2 pHost 2 Virtual Underlay VM2 Container D Container E Container F Linux kernel routing (no encapsulation) pNIC vNIC vNIC veth0 veth1 veth2 Physical Underlay
- 15. Project Calico is sponsored by@projectcalico Variant: 1 vm per host, no virtual underlay, straight-up IP pHost 1 pNIC vNIC VM1 Container A Container B Container C Linux kernel routing (no encapsulation) veth0 veth1 veth2 pHost 2 VM2 Container D Container E Container F Linux kernel routing (no encapsulation) pNIC vNIC veth0 veth1 veth2 Physical Underlay
- 16. Project Calico is sponsored by@projectcalico Results: bare metal performance from virtual networks 0 1 2 3 4 5 6 7 8 9 10 Bare metal Calico OVS+VXLAN Throughput Gbps 0 20 40 60 80 100 120 Bare metal Calico OVS+VXLAN CPU % per Gbps Source: https://www.projectcalico.org/calico-dataplane-performance/
- 17. Project Calico is sponsored by@projectcalico Some container frameworks still assume port mapping E.g. Marathon load balancer service (but being fixed…) Some PaaS’s not yet supporting IP per container But several moving to build on Kubernetes, and will likely pick it up Gotcha #5: IP per container not yet universally supported
- 18. Project Calico is sponsored by@projectcalico You can easily get your configuration wrong and get sub- optimal performance, e.g. select wrong Flannel back-end for your fabric turn off AWS src-dest IP checks get MTU size wrong for the underlay… Gotcha #6: running on public cloud
- 19. Project Calico is sponsored by@projectcalico Consequences of MTU size… 0 50 100 150 200 250 300 t2.micro m4.xlarge qperf bandwidth Bare Metal Calico
- 20. Project Calico is sponsored by@projectcalico Consequences of MTU size… 0 50 100 150 200 250 300 t2.micro m4.xlarge qperf bandwidth Bare Metal Calico (MTU=1440) Calico (MTU=8980)
- 21. Project Calico is sponsored by@projectcalico Suppose we assign a /24 per Kubernetes node (=> 254 pods) Run 10 VMs per server, each with a Kubernetes node 40 servers per rack 20 racks per data center 4 data centers => now need a /15 for the rack, a /10 space for the data center, and the entire 10/8 rfc1918 range to cover 4 data centers. … and hope your business doesn’t expand to need a 5th data center! Gotcha #7: IP addresses aren’t infinite
- 22. Project Calico is sponsored by@projectcalico Kubernetes CNI fairly stable Fine-grained policy being added – will move from alpha (annotation— based) to first-class citizen API Mesos – multiple ways to network your container Net-modules – but only supports Mesos containerizer Docker networking – but then not fully integrated e.g. into MesosDNS CNI – possible future, but not here today Roll-your-own orchestrator-network co-ordination – the approach some of our users have taken Docker Swarm / Docker Datacenter still early; libnetwork evolution? policy? Gotcha #8: orchestration platforms support still evolving
- 23. Project Calico is sponsored by@projectcalico Docker libnetwork provides limited functionality / visibility to plug-ins E.g. network name you specify as a user is NOT passed to the underlying SDN Consequences: Diagnostics hard to correlate Hard to enable ”side loaded” commands referring to networks created on Docker command line (e.g. Calico advanced policy) Hard to network between Docker virtual network domain and non- containerized workloads Gotcha #9: Docker libnetwork is “special”
- 24. Project Calico is sponsored by@projectcalico “Can you write a function that tells me when all nodes have caught up to the global state?” Sure… Gotcha #10: at cloud scale, nothing ever converges function is_converged() return false
- 25. Project Calico is sponsored by@projectcalico
Editor's Notes
- #13: All the containers can talk to one another Things like Kubeproxy will allow a single service VIP to access the containers inside the virtual network But some legacy apps need direct access to the containers, and there’s no on/off-ramp possible