Weird things we've seen with OpenStack Neutron
0 likes726 views
The document discusses common issues encountered with OpenStack Neutron, including orphaned namespaces, duplicate segmentation IDs, and duplicate routers, along with troubleshooting techniques. It provides insights into the architecture of Neutron, typical user complaints, and specific steps for diagnosing network problems. Various commands for monitoring and managing network components are also shared to assist users in resolving connectivity issues.
Weird things we've seen with OpenStack Neutron
- 1. Weird stuff we've seen with OpenStack Neutron (And what to do about it)
- 3. OpenStack Neutron • So$ware-defined networking component • Users define their own virtual networks • Manages IP address assignment • Floa?ng IP addresses • Supports many different back-ends - OpenvSwitch, VMware NSX, Cisco UCS, Midokura....
- 4. Neutron usage1 1 Source: OpenStack User Survey, October 2015
- 6. Architecture, con-nued • neutron-{server,agent} • OpenvSwitch • Linux bridging • Linux network namespaces • L2 • L3
- 7. Namespaces • L2 namespace • DHCP • L3 namespace • Rou4ng • NAT • Metadata
- 8. Common problems - typical user complaints • VM can't obtain an IP address • Can't ping / connect to my VM • Intermi9ent connec:vity
- 9. Weirdness #1 - orphaned namespaces • Default (on Ubuntu) is not to delete namespaces at all (!) • Bug in iproute2 package • h=ps://bugs.launchpad.net/neutron/+bug/1052535 • Misconfigured sudo rules meant that network namespaces weren't being deleted • Mismatch between interfaces configured in a namespace and what Neutron expects
- 10. Finding out what's supposed to be where for netnode in osnet{0..4} ; do echo $netnode for router in $(ssh $netnode 'ip netns list | grep qrouter | cut -d - -f 2-20') ; do neutron router-show $router | grep -i unable done done Then delete each invalid namespace and associated OVS port. • Pro%p: Don't run neutron-ovs-cleanup!
- 11. Weirdness #2 - duplicate segmenta4on ID • Customer support ,cket with instances unable to obtain an IP via DHCP • Some serious digging required...
- 12. Tracing packet flows • tcpdump on compute node and in network namespaces • Packets not always arriving where you'd expect • Have to look at OpenFlow rules
- 13. DHCP agent neutron dhcp-agent-list-hosting-net 4dc325ed-f141-41d9-8d0a-4f513defacad +--------------------------------------+--------+----------------+-------+ | id | host | admin_state_up | alive | +--------------------------------------+--------+----------------+-------+ | 1beb99ef-e6f6-4083-8fb6-661f2f61c565 | osnet1 | True | :-) | +--------------------------------------+--------+----------------+-------+ neutron net-show -F provider:segmentation_id 4dc325ed-f141-41d9-8d0a-4f513defacad +--------------------------+-------+ | Field | Value | +--------------------------+-------+ | provider:segmentation_id | 11 | +--------------------------+-------+ • 11 in hex = 0xb
- 14. root@osnet1:~# ovs-ofctl dump-flows br-tun table=2 NXST_FLOW reply (xid=0x4): cookie=0x0, duration=875584.823s, table=2, n_packets=85, n_bytes=10880, idle_age=11560, hard_age=65534, priority=1,tun_id=0x14 actions=mod_vlan_vid:43,resubmit(,10) cookie=0x0, duration=2578615.436s, table=2, n_packets=1345, n_bytes=128202, idle_age=27174, hard_age=65534, priority=1,tun_id=0x10 actions=mod_vlan_vid:2,resubmit(,10) cookie=0x0, duration=2578611.677s, table=2, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=1,tun_id=0xd actions=mod_vlan_vid:12,resubmit(,10) cookie=0x0, duration=1806356.959s, table=2, n_packets=5140, n_bytes=364533, idle_age=341, hard_age=65534, priority=1,tun_id=0x21 actions=mod_vlan_vid:35,resubmit(,10) cookie=0x0, duration=2578610.661s, table=2, n_packets=1035919, n_bytes=180430025, idle_age=65534, hard_age=65534, priority=1,tun_id=0x11 actions=mod_vlan_vid:16,resubmit(,10) cookie=0x0, duration=1465355.359s, table=2, n_packets=418252, n_bytes=81112777, idle_age=52, hard_age=65534, priority=1,tun_id=0x13 actions=mod_vlan_vid:42,resubmit(,10) cookie=0x0, duration=1631281.273s, table=2, n_packets=445, n_bytes=52848, idle_age=65534, hard_age=65534, priority=1,tun_id=0x17 actions=mod_vlan_vid:37,resubmit(,10) cookie=0x0, duration=2578609.671s, table=2, n_packets=1821, n_bytes=167272, idle_age=16439, hard_age=65534, priority=1,tun_id=0xc actions=mod_vlan_vid:17,resubmit(,10) cookie=0x0, duration=2574619.932s, table=2, n_packets=490592856, n_bytes=279835052124, idle_age=65534, hard_age=65534, priority=1,tun_id=0x19 actions=mod_vlan_vid:19,resubmit(,10) cookie=0x0, duration=2578613.06s, table=2, n_packets=18, n_bytes=756, idle_age=65534, hard_age=65534, priority=1,tun_id=0xe actions=mod_vlan_vid:8,resubmit(,10) cookie=0x0, duration=1469974.534s, table=2, n_packets=6992536, n_bytes=1567235429, idle_age=9, hard_age=65534, priority=1,tun_id=0x7 actions=mod_vlan_vid:41,resubmit(,10) cookie=0x0, duration=2144082.193s, table=2, n_packets=2583, n_bytes=461773, idle_age=65534, hard_age=65534, priority=1,tun_id=0x1d actions=mod_vlan_vid:32,resubmit(,10) cookie=0x0, duration=2578611.169s, table=2, n_packets=4230304, n_bytes=917966422, idle_age=0, hard_age=65534, priority=1,tun_id=0x5 actions=mod_vlan_vid:14,resubmit(,10) cookie=0x0, duration=85135.825s, table=2, n_packets=1739, n_bytes=130092, idle_age=65534, hard_age=65534, priority=1,tun_id=0x1f actions=mod_vlan_vid:53,resubmit(,10) cookie=0x0, duration=979.195s, table=2, n_packets=123, n_bytes=11895, idle_age=933, priority=1,tun_id=0x22 actions=mod_vlan_vid:54,resubmit(,10) cookie=0x0, duration=1898543.732s, table=2, n_packets=240, n_bytes=30712, idle_age=65534, hard_age=65534, priority=1,tun_id=0x16 actions=mod_vlan_vid:34,resubmit(,10) cookie=0x0, duration=2578614.004s, table=2, n_packets=5595775, n_bytes=5465543420, idle_age=4, hard_age=65534, priority=1,tun_id=0x8 actions=mod_vlan_vid:6,resubmit(,10) cookie=0x0, duration=1473941.345s, table=2, n_packets=4202494, n_bytes=2516931444, idle_age=9, hard_age=65534, priority=1,tun_id=0x4 actions=mod_vlan_vid:40,resubmit(,10) cookie=0x0, duration=2578619.787s, table=2, n_packets=103506, n_bytes=13925984, idle_age=0, hard_age=65534, priority=0 actions=drop wat.
- 16. Missing OpenFlow rule root@osnet1:~# ovs-ofctl dump-flows br-tun table=2 | grep 0xb root@osnet1:~# echo $? 1 Try to re-add that network to the responsible agent: $ neutron dhcp-agent-network-remove 1beb99ef-e6f6-4083-8fb6-661f2f61c565 4dc325ed-f141-41d9-8d0a-4f513defacad Removed network 4dc325ed-f141-41d9-8d0a-4f513defacad from DHCP agent $ neutron dhcp-agent-network-add 1beb99ef-e6f6-4083-8fb6-661f2f61c565 4dc325ed-f141-41d9-8d0a-4f513defacad Added network 4dc325ed-f141-41d9-8d0a-4f513defacad to DHCP agent root@osnet1:~# ovs-ofctl dump-flows br-tun table=2 | grep 0xb cookie=0x0, duration=0.945s, table=2, n_packets=14, n_bytes=588, idle_age=0, priority=1,tun_id=0xb actions=mod_vlan_vid:55,resubmit(,10)
- 17. Weirdness #3 - duplicate routers • Intermi)ent connec-vity issues groan • No DVR or L3-HA enabled • Routers scheduled and created twice on two network nodes • Same network configura-on in each namespace
- 18. Duplicate routers › neutron l3-agent-list-hosting-router fe79ae7e-debf-44b9-8fd7-601abd5fb928 +--------------------------------------+--------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+--------+----------------+-------+----------+ | 48132c36-b6b1-40fa-b9d9-5474f4f27c3a | osnet0 | True | :-) | | | c821a370-b301-40c5-8b7b-25d147ffc904 | osnet1 | True | :-) | | +--------------------------------------+--------+----------------+-------+----------+ › neutron router-show fe79ae7e-debf-44b9-8fd7-601abd5fb928 +-----------------------+----------------------------------+ | Field | Value | +-----------------------+----------------------------------+ | admin_state_up | True | | distributed | False | | ha | False | | status | ACTIVE | | tenant_id | 7d718c99276c43d1992d64d061d98f15 | +-----------------------+----------------------------------+
- 19. How to approach troubleshoo0ng Troubleshoo*ng checklist • UUIDs for instance, loca2on, MAC address • UUIDs for network, subnet, router • Network node hos2ng L2 and L3 agents
- 20. Useful commands - neutron $ neutron agent-list $ neutron l3-agent-list-hosting-router $router_uuid $ neutron dhcp-agent-list-hosting-net $net_uuid $ neutron router-list-on-l3-agent $agent_uuid $ neutron net-list-on-dhcp-agent $net_uuid $ neutron help
- 21. Useful commands - OpenvSwitch $ ovs-vsctl show $ ovs-ofctl dump-flows $bridge $ ovs-dpctl show
- 22. (More) useful commands Standard network troubleshoo1ng toolkit: $ tcpdump -enl -i eth1 | grep -i dhcp $ ip netns exec $netns tcpdump port 67 or port 68 -lne $ ip route $ ip address $ iptables-save $ brctl $ mtr Etc.