Open stackdaykorea2016 wedge
1 like499 views
The document discusses the experience of managing Broadcom-based network switches using Open Network Linux (ONL) during the OpenStack Day Korea 2016. It covers the Open Compute Project (OCP), Facebook's Wedge hardware, and details on L2 switching and L3 routing using OpenNSL, providing insights into networking hardware and software disaggregation. Additionally, it includes implementation examples and usage cases for network operations, specifically focusing on open-source contributions to enhance data center networking efficiency.
Open stackdaykorea2016 wedge
- 1. OpenNSL으로 브로드콤 기반 네트워크 스위치 제어하기 OpenStack Day Korea 2016 NIC 기술원 SDI Tech. Lab Manager 서준호 2016. 2. 18
- 2. 1 Contents 본 발표는 지난 한달 간 Facebook이 제공한 Wedge를 가지고 이리 저리 가지고 논 경험을 담고 있음 1. OCP 소개 2. Facebook Wedge H/W 소개 3. Networking Operating System – Open Network Linux (ONL) 4. Broadcom Trident2 & OpenNSL 소개 5. OpenNSL 튜토리얼 (L2 스위칭, L3 라우팅)
- 3. 2 OCP 소개 OCP (Open Compute Project) § Facebook 주도의 데이터센터 인프라 설계 기술 공유를 위한 오픈 하드웨어 프로젝트 §Facebook 은 Oregon 주 Prineville 데이터센터 인프라 스펙을 공개 §IT 기술을 접목시켜 경제적이고 효율적 관리를 통해 $20억 절감 § 프로젝트 §Server/Storage/Networking/Open Rack/Data Center/Management § SKT 는 올해 2월 가입 §OCP Summit 2016 (3월) 에서 발표 예정
- 4. 3 OCP Networking 소개 Project Chairs: Omar Baldonado | Carlos Cardenas 네트워크 하드웨어 + 네트워킹 소프트웨어 § Disaggregation + Open § 현재는 ToR 스위치와 Spine 스위치 스펙 정의에 집중하고 있음 § 또한 각 칩 벤더간의 서로 다른 Hardware Abstract Layer (HAL) 영역을 하나의 공통된 인터페이스 Service Abstract Interface (SAI) 로 스펙화 진행 중
- 5. 4 Facebook이 개발한 1RU(Rack Unit) 크기의 ToR Switch Facebook Wedge 소개 (1/3) Modular Design § Micro-server module §Intel Avoton C2550 (Atom) § Main switch module §16 QSFP §4x Fan module § BCM56850 (Trident2) § Open BMC module Spine Switch
- 6. 5 Facebook Wedge 소개 (2/3) High Level Block Diagram 기존 서버 아키텍처와 다를게 없음!
- 7. 6 High Speed Interface Connectivity § 16x QSFP (each x4 SerDes 10GbE) § 16x (not used), but for 6-Pack Facebook Wedge 소개 (3/3) Main Switching ASIC - BCM56850 (Trident2) 2U Yosemite Wedge 16 x Yosemite
- 8. 7 Power on & Boot up Wedge ONIE: Boot Loader contributed by Cumulus Networks § GRUB2 (x86) or U-Boot (powerpc or other architectures) § Reinstall/Uninstall/Rescue/Boot NOSes on OCP bare metal switches over IPv6 and HTTP
- 9. 8 Installing Networking Operating System Switch Light Current ONIE compatible NOS List We choose Open Network Linux (ONL) § OCP 커뮤니티에서 베어 메탈 스위치를 위한 레퍼런스 네트워크 운영체제 § Big Switch가 OCP 커뮤니티 네트워킹 그룹에 오픈소스 기여 § 데비안 리눅스 기반 + 네트워크 ASIC 칩 드라이버 + ɑ (vendor specific) § 현재 Big Switch Networks, 페이스북, NTT 사용 중 § 현재 베어 메탈 스위치 제조 벤더들이 지원 à 앞으로도 계속 ONL이 지원되는 OCP 표준 스위치 증가할 예정 (OCP summit `15)
- 10. 9 Installing Networking Operating System
- 11. 10 네트워크 스위치가 되기 위해서는… 지금까지 한 일 네트워크 스위치가 되기 위해서 할 일 Wedge
- 12. 11 Facebook - the next-generation Facebook data center network Wedge (ToR 스위치) 6-pack (Spine스위치) Clos networks FBOSS Facebook ONL Use Case
- 13. 12 Use Cases: NTT NTT’s EVPN NTT ONL Use Case
- 14. 13 OpenNSL 소개 Broadcom Switching ASIC (i.e., Trident+, Trident2) Chip 제어를 위한 오픈소스 기반 Hardware Abstraction Layer (HAL) Licence Policy § OEM/ODM Development Package § Community Development Package Features § L2/L3/VLAN management § Port and switch management § Link monitoring § Packet transmit and receive § Trunking § … Supported Platforms § Acction Wedge, AS5712 § Broadcom reference platform (SVK)
- 15. 14 OpenNSL 소개 Broadcom Switching ASIC (i.e., Trident+, Trident2) Chip 제어를 위한 오픈소스 기반 Hardware Abstraction Layer (HAL) OpenNSL Use Cases § FBOSS (Facebook 주도) § Open Network Linux (Big Switch 주도) § BroadView § Open L3 Routing (NTT 주도) § OpenSwitch (HP주도)
- 16. 15 MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ Input Arbiter (PacketHeader Parser) PacketProcessing Pipeline MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ Output Queues High Level Block Diagramof BroadcomASIC OpenNSL 소개 PCI-E Kernel SpaceUser Space OpenNSL Broadcom SDK Kernel BDE Kernel KNET DMA eth0 eth1 eth2 SRAM TCAM DRAM
- 17. 16 Example – L2 Switching Basic Three Operations of L2 Switch § Address learning §Src MAC matching §Automatically learning § L2 Forwarding § Address ageing MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ Input Arbiter (PacketHeader Parser) Address Resolution Logic (ARL) MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ Output Queues ARL Table Memory MAC | VLAN | PORT | TIME opennsl_l2_addr_add(mac, vlan, port, flags); flags: OPENNSL_L2_DISCARD_SRC OPENNSL_L2_DISCARD_DST OPENNSL_L2_L3LOOKUP OPENNSL_L2_MOVE_PORT OPENNSL_L2_COPY_TO_CPU CALLBACK … opennsl_l2_addr_add() opennsl_l2_addr_delete() opennsl_l2_addr_get() opennsl_l2_addr_register() opennsl_l2_addr_t_init()
- 18. 17 MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ Input Arbiter (PacketHeader Parser) ARL MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ Output Queues Basic Three Operations of L2 Switch § Address learning § L2 Forwarding §Dst MAC matching §Default - Flooding § Address ageing Example – L2 Switching Forwarding Logic Forwarding Table Memory MAC | MAC_MASK opennsl_l2_station_add(mac, mask, flags); flags: OPENNSL_L2_STATION_ARP_RARP OPENNSL_L2_STATION_IPV4 opennsl_l2_station_add() opennsl_l2_station_delete() opennsl_l2_station_get() opennsl_l2_station_t_init()
- 19. 18 Example – L3 Routing L3 Routing § Routing between different sub-networks § Longest Prefix Matching (LPM) 192.168.1.0/24 192.168.2.0/24 192.168.3.0/24 Subnet 1 Subnet 2 Subnet 3 192.168.1.1 192.168.2.1 192.168.3.1
- 20. 19 Example – L3 Routing MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ Input Arbiter (PacketHeader Parser) L2 Logic MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ Output Queues L3 Logic L3 Routing 1. Set L3 egress mode 2. Create IP interfaces 3. Make the address learn on a VLAN and port 4. Create egress object 5. Add host point 6. Add default route opennsl_switch_control_set(opennslSwitchL3EgressMode);
- 21. 20 Example – L3 Routing MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ Input Arbiter (PacketHeader Parser) L2 Logic MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ Output Queues L3 Logic L3 Routing 1. Set L3 egress mode 2. Create Ingress/Egress IP interfaces 3. Make the address learn on a VLAN and port 4. Create egress object 5. Add host point 6. Add default route opennsl_l3_intf_create(vlan, mac, in_port); opennsl_l3_intf_create(vlan, mac, out_port);
- 22. 21 Example – L3 Routing MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ Input Arbiter (PacketHeader Parser) L2 Logic MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ Output Queues L3 Logic L3 Routing 1. Set L3 egress mode 2. Create Ingress/Egress IP interfaces 3. Make the address learn on a VLAN and port 4. Create egress object 5. Add host point 6. Add default route opennsl_l2_addr_addr(vlan, in_port, mac); opennsl_l2_addr_addr(vlan, out_port, mac); flags |= OPENNSL_L2_L3LOOKUP ARL Table Memory MAC | VLAN | PORT | TIME
- 23. 22 Example – L3 Routing MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ Input Arbiter (PacketHeader Parser) L2 Logic MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ Output Queues L3 Logic L3 Routing 1. Set L3 egress mode 2. Create Ingress/Egress IP interfaces 3. Make the address learn on a VLAN and port 4. Create egress object 5. Add host point 6. Add default route opennsl_l3_egress_create(vlan, out_port, out_intf, nexthop_mac); L3 Forwarding Table out_inf | nexthop_mac
- 24. 23 Example – L3 Routing MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ Input Arbiter (PacketHeader Parser) L2 Logic MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ Output Queues L3 Logic L3 Routing 1. Set L3 egress mode 2. Create Ingress/Egress IP interfaces 3. Make the address learn on a VLAN and port 4. Create egress object 5. Add host point 6. Add default route opennsl_l3_host_add(host_ip) L3 Forwarding Table host_ip | out_inf | nexthop_mac
- 25. 24 Example – L3 Routing MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ MAC RxQ CPU RxQ Input Arbiter (PacketHeader Parser) L2 Logic MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ MAC TxQ CPU TxQ Output Queues L3 Logic L3 Routing 1. Set L3 egress mode 2. Create Ingress/Egress IP interfaces 3. Make the address learn on a VLAN and port 4. Create egress object 5. Add host point 6. Add default route opennsl_l3_route_add(default_ip, out_inf) L3 Forwarding Table host_ip | out_inf | nexthop_mac default_ip | out_inf | nexthop_mac
- 26. 25 Thanks 감사합니다.
- 27. 26 Example – L2 Switching Basic Three Operations of L2 Switch § Address learning – Src MAC matching § L2 Forwarding § Address ageing Host A Host B 00:00:0A 00:00:0B 1 MAC ADD VLAN PORT 00:00:0A 1 1 00:00:0B 1 1 00:00:0C 1 2 00:00:0D 1 2 Address Learning Table Host C 00:00:0C 00:00:0D Host D L2 Switch opennsl_l2_addr_t addr; opennsl_l2_addr_t_init(&addr, mac, vlan); addr.flags |= SOME_ACTION; opennsl_l2_addr_add(&addr); 1 2 VLAN OPENNSL_L2_DISCARD_SRC OPENNSL_L2_DISCARD_DST OPENNSL_L2_COPY_TO_CPU …
- 28. 27 Example – L2 Switching Basic Three Operations of L2 Switch § Address learning § L2 Forwarding (Bridging) – Dst MAC matching § Address ageing Host A Host B 00:00:0A 00:00:0B 1 MAC ADD PORT VLAN 00:00:0A 1 1 00:00:0B 1 1 00:00:0C 2 2 00:00:0D 2 2 Address Learning Table Host C 00:00:0C 00:00:0D Host D L2 Switch1 2 VLAN1 VLAN2 opennsl_l2_station_t sta; opennsl_l2_station_t_init(&sta); sta.dst_mac = 00:00:0D sta.dst_mac_mask = FF:FF:FF opennsl_l2_station_add(&sta); MAC ADD 00:00:00D Station Table
- 29. 28 Example – L3 Routing Routing between different networks Host A Host B 00:00:0A 00:00:0B 1 Host C 00:00:0C 00:00:0D Host D L3 Router1 2 Net1 Net2 IP ADDR Intf 00:00:0A 1 00:00:0B 1 00:00:0C 2 00:00:0D 2
- 30. 29 Example – L3 Routing Routing between different networks 1. Set L3 egress mode 2. Create IP interfaces 3. Make the address learn on a VLAN and port 4. Create egress object 5. Add host point 6. Add default route opennsl_switch_control_set(opennslSwitchL3EgressMode); L3 Routing Logic
- 31. 30 Example – L3 Routing Routing between different networks 1. Set L3 egress mode 2. Create ingress & egress IP interfaces 3. Make the address learn on a VLAN and port 4. Create egress object 5. Add host point 6. Add default route opennsl_l3_intf_create(vlan, mac, in_port); opennsl_l3_intf_create(vlan, mac, out_port); L3 Routing Logic In_intf out_intfin_port out_port
- 32. 31 Example – L3 Routing Routing between different networks 1. Set L3 egress mode 2. Create ingress & egress IP interfaces 3. Make the address learn on a VLAN and port 4. Create egress object 5. Add host point 6. Add default route opennsl_l2_addr_addr(vlan, in_port, mac); opennsl_l2_addr_addr(vlan, out_port, mac); flags |= OPENNSL_L2_L3LOOKUP L3 Routing Logic In_intf out_intfin_port out_port
- 33. 32 Example – L3 Routing Routing between different networks 1. Set L3 egress mode 2. Create ingress & egress IP interfaces 3. Make the address learn on a VLAN and port 4. Create egress object 5. Add host point 6. Add default route opennsl_l3_egress_create(vlan, out_port, out_intf, nexthop_mac); L3 Routing Logic In_intf out_intfin_port out_portEgress Object
- 34. 33 Example – L3 Routing Routing between different networks 1. Set L3 egress mode 2. Create ingress & egress IP interfaces 3. Make the address learn on a VLAN and port 4. Create egress object 5. Add host point 6. Add default route opennsl_l3_host_add() L3 Routing Logic In_intf out_intfin_port out_portEgress Object IP ADDR Intf HOST_A out_intf Routing Table
- 35. 34 Example – L3 Routing Routing between different networks 1. Set L3 egress mode 2. Create ingress & egress IP interfaces 3. Make the address learn on a VLAN and port 4. Create egress object 5. Add host point 6. Add default route opennsl_l3_route_add() L3 Routing Logic In_intf out_intfin_port out_portEgress Object IP ADDR Intf HOST_A out_intf DEFAULT out_intf Routing Table