Baker: Scaling OVN with Kubernetes API Server
1 like383 views
This document discusses using Kubernetes API server framework (Baker) to scale out OVN control plane. It aims to distribute OVN northd and central OVSDB cluster. Challenges include lack of clustering in OVSDB SB and distributed state management. Baker proposes distributed northd, scale-out central cluster backed by ETCD, and distributed agents to watch local objects. This would address connectivity issues in large cloud deployments. The document also discusses addressing segmentation vs connectivity, using ACLs and address sets for L3 segmentation, and evaluating port security and ACL processing on each hypervisor.
Baker: Scaling OVN with Kubernetes API Server
- 1. Baker: Scaling OVN with Kubernetes API Server Han Zhou OpenStack Summit Boston 2017
- 2. Why OVN? OVS is GREAT. OVN makes it GREATER! 2
- 3. OVN Challenges ● OVN is distributed, but not fully … ○ Can we distributed Northd? 3 Northd NB SB OVN-Controller OVS HV HV … Central OVSDB HV
- 4. OVN Challenges ● OVSDB SB ○ No clustering (yet) 4 Northd NB SB OVN-Controller OVS HV HV … Central OVSDB HV It is nothing but distributed state management ...
- 5. Scale-out with Baker ● Distributed northd ○ Computes lflows for local only ● Scale-out central cluster ○ K8S API server framework ○ Backed by ETCD ○ Clustering ● Distributed agents ○ Watch for local objects only ○ Translate objects to NB DB 5 Northd NB SB OVN-Controller OVSHV Central ETCD ETCD ETCDBaker API server Baker API server Baker API server Baker Agent HV … HV RESTful API
- 6. One more thing ... 6 â—Ź Northd and ovn-controller are all distributed â—Ź They process data related to local HV only But what does this mean?
- 7. In terms of overlay ... 7 ● Logical-to-physical mapping states (port-binding) for connectivity ● Doesn’t scale when everyone talks to everyone else in a *large* zone ○ Maybe not the case for public cloud, or small-to-medium enterprise cloud. ○ But it is typical use case for eBay’s private cloud.
- 8. Are we solving the right problem? 8 â—Ź Connectivity v.s. Segmentation â—Ź L2 Segmentation v.s. L3 segmentation â—Ź Address sets (L3) based segmentation â—‹ ACL: default deny, whitelist access â—‹ IPAM: â– Use ip efficiently â– Summarized CIDRs to reduce address set size
- 9. Flat network 9 ● Reuse OVN abstraction and pipeline ○ Port security ○ ARP proxy ○ ACL ○ LB ○ … ○ But NOT overlay ● Use localnet port to connect to physical network directly ● Data to be processed by each HV depends on size of AddressSet used by ACLs that apply to ports on the HV
- 10. Baker Object Model â—Ź Similar as OVN NB Schema â—‹ Logical Port â– Addresses â– Port security â—‹ ACL â—‹ Address Set â—‹ Load balancer (TBD) â—‹ ... â—Ź Differences â—‹ No Logical Switch (local) â—‹ Port-SecGroup binding â—‹ ACL: SecGroup instead of individual ports in inport/outport 10
- 11. Neutron Plugin â—Ź Support security group, with API extensions â—‹ Address set - support external IPs from legacy systems â—‹ Security group rule packet logging 11
- 12. Scalability - Control plane throughput 12 â—Ź Test â—‹ E2E: Neutron - Baker - OVS â—‹ Simulated 1k HVs on 10 BMs â– OVS/OVN 2.7 â—‹ 1 node Neutron + mysql â—‹ 1 node Baker API server + ETCD â– K8s 1.6 pre-release, etcd 3.0 â—Ź Result for single client (parallel test TBD) â—‹ Result impacted by SG (address set) size â—‹ ~3 ports/sec for SG size 1K
- 13. Scalability - Latency 13 â—Ź Test â—‹ E2E from Neutron to OVS flow installation for the port created â– Create port from neutron, bind port in ovs on HV â– Wait: â—Ź ovn-nbctl wait-until Logical_Switch_Port <port> up=true â—Ź ovn-nbctl --wait=hv sync â—‹ Create ports on top of existing 10K ports, 1K HVs, SG size 1K â—‹ 10K * 3 (flows/ACL) = 30K flows / ovs port â—Ź Result â—‹ Avg 2 sec
- 14. Improvement - ovn-controller 14 â—Ź Flow computation blocks flow installation â—Ź Improvement: avoid repeated computation when in-flight messages to OVS pending â—Ź Test result (SG size 10k, flow installation for 10 ports on HV): â—‹ 10k * 3 * 10 = 300k OVS flows â—‹ Before: 50 min â—‹ After: 16 sec
- 15. Other Lessons learned 15 ● Postpone ACL expanding from Neutron to HV ○ Introduce port-group binding object in Baker ○ Use port-group instead of lport in “inport/outport” in ACL ○ Baker agent expand ACL on HV for local lports only ○ Benefit: ■Reduced Neutron overhead ■Reduced API calls from Neutron to Baker ■Reduced data size in Baker
- 16. Other Lessons learned 16 â—Ź Baker RESTful API: use Protobuf instead of JSON-RPC â—‹ 10 - 20 % throughput increase for SG size 1k - 10k â—‹ Lower CPU cost on API-server
- 17. Thanks! Q & A