DoS and DDoS mitigations with eBPF, XDP and DPDK
2 likes1,419 views
The document compares eBPF, XDP and DPDK for packet inspection. It describes the speaker's experience using these tools to build a virtual machine that can handle 10Gbps of traffic and drop packets to mitigate DDoS attacks. It details how eBPF and XDP were able to achieve higher packet drop rates than iptables or a custom module. While DPDK could drop traffic at line rate, it required specialized hardware and expertise. Ultimately, XDP provided the best balance of performance, driver support and programmability using eBPF to drop millions of packets per second.
DoS and DDoS mitigations with eBPF, XDP and DPDK
- 1. Comparison of eBPF, XDP and DPDK for packet inspection Marian Marinov <mm@yuhu.biz> Chief System Architect of SiteGround Linux Piter 2019
- 2. Who am I?Who am I? ❖❖ Who am I?Who am I? - Chief System Architect of SiteGround.com- Chief System Architect of SiteGround.com - Sysadmin since 1996- Sysadmin since 1996 - Organizer of OpenFest, BG Perl- Organizer of OpenFest, BG Perl Workshops, LUG-BG, RailsGirls and othersWorkshops, LUG-BG, RailsGirls and others - Teaching Network Security and Linux- Teaching Network Security and Linux System Administration courses in SofiaSystem Administration courses in Sofia University and SoftUniUniversity and SoftUni
- 3. Why do we need this?Why do we need this? Frequency of DoS/DDoS attacks to ourFrequency of DoS/DDoS attacks to our infrastructureinfrastructure ➢ 4-10 Gbps 6-8 times a month4-10 Gbps 6-8 times a month ➢ 10-40 Gbps maybe 2-3 times a month10-40 Gbps maybe 2-3 times a month ➢ 100+ Gbps around 2 times a month100+ Gbps around 2 times a month
- 4. More statsMore stats Attacks resulting in service degradation:Attacks resulting in service degradation: ➢ for the past 276 days we had 31 DDoS attacks ➢ some of the months, no attackssome of the months, no attacks ➢ but some months, up to 9but some months, up to 9 ➢ 2019 – 31 attacks2019 – 31 attacks ➢ 2018 – 75 attacks2018 – 75 attacks ➢ 2017 – 69 attacks2017 – 69 attacks ➢ 2016 – 84 attacks2016 – 84 attacks Note: I have manually counted the e-mails. The numbers can be slightly inaccurate.Note: I have manually counted the e-mails. The numbers can be slightly inaccurate.
- 5. Most attacks are basicMost attacks are basic ➢ 20k pps toward ISC Bind can consume20k pps toward ISC Bind can consume up to 30 CPU coresup to 30 CPU cores ➢ a child can generate that on its laptop, at homea child can generate that on its laptop, at home
- 6. General solutionsGeneral solutions ➢ Buy additional bandwidthBuy additional bandwidth ➢ Buy a very expensive scrubbing deviceBuy a very expensive scrubbing device OROR ➢ Offload this task to other companies, likeOffload this task to other companies, like CloudFlareCloudFlare
- 7. Hosted solution issuesHosted solution issues ➢ Not every DataCenter is wiling to invest inNot every DataCenter is wiling to invest in these devicesthese devices ➢ Shared devicesShared devices ➢ Attacks can be larger then the capacity of theAttacks can be larger then the capacity of the devicedevice ➢ Larger attacks almost always result in null routeLarger attacks almost always result in null route ➢ Attacks saturating the uplinks can affect otherAttacks saturating the uplinks can affect other machines in the rack and/or rowmachines in the rack and/or row
- 8. Cloud solution issuesCloud solution issues ➢ You have to point your DNS to the serviceYou have to point your DNS to the service providerprovider ➢ Controlling your DNS is now only API basedControlling your DNS is now only API based ➢ Large DNS updates become an issueLarge DNS updates become an issue ➢ Not suitable for hosting companiesNot suitable for hosting companies
- 9. Requirements?Requirements? ➢ Build a VM that can handle 10Gbps withBuild a VM that can handle 10Gbps with ~8Mpps~8Mpps ➢ Why a VM?Why a VM? ➢ scrub UDP DNS and NTP trafficscrub UDP DNS and NTP traffic ➢ scrub TCP traffic by implementing SYN cookiesscrub TCP traffic by implementing SYN cookies ➢ scrub all unrelated trafficscrub all unrelated traffic ➢ cache HTTP responses(wishful thinking) :)cache HTTP responses(wishful thinking) :)
- 10. Linux Network FlowLinux Network Flow
- 11. Linux Network FlowLinux Network Flow
- 12. Linux Network FlowLinux Network Flow
- 13. Linux Network FlowLinux Network Flowreceived packet XDP eBPF alloc_skb ingress (qdisc) bridge check broute brouting bridge check nat prerouting raw prerouting mangle forward nat prerouting bridging decision filter forward filter input routing decision filter forward mangle prerouting mangle forward filter forward mangle forward nat postrouting mangle postrouting mangle postrouting nat postrouting nat postrouting conntrack nat output filter output nat postrouting bdrige netfilter egress (qdisc) other net
- 14. 10M packet drop10M packet drop ➢ in 2018 CloudFlare published the article:in 2018 CloudFlare published the article: How to drop 10m packetsHow to drop 10m packets ➢ I confirm their results with a few additions:I confirm their results with a few additions: ➢ iptables can drop at best 2m ppsiptables can drop at best 2m pps Note: with only one entry in the PREROUTING chain of the mangle tableNote: with only one entry in the PREROUTING chain of the mangle table ➢ heaving multiple entries in that chain easilyheaving multiple entries in that chain easily becomes a problembecomes a problem ➢ even if you use ipset with that, you have a bigeven if you use ipset with that, you have a big problem when updating that informationproblem when updating that information
- 17. 10M packet drop10M packet drop ➢ CloudFlare demo code can be found on GitHub
- 18. So, how I started?So, how I started? ➢ I already knew about XDPI already knew about XDP ➢ But I decided to be “smart ass” and wrote anBut I decided to be “smart ass” and wrote an iptables module...iptables module... ➢ It could handle between 260k and 280k ppsIt could handle between 260k and 280k pps
- 19. Not good enough... eBPFNot good enough... eBPF ➢ I also knew I can use eBPF for that...I also knew I can use eBPF for that... ➢ from the talk of Daniel Borkmann fromfrom the talk of Daniel Borkmann from FOSDEM 2016FOSDEM 2016 ➢ It was better, but not enough...It was better, but not enough... ➢ 320-350k pps drop rate320-350k pps drop rate ➢ with 2000 domains and UDP packet checkingwith 2000 domains and UDP packet checking ➢ no checksums thouno checksums thou
- 20. DPDKDPDK ➢ I had previous experience with DPDK ➢ So I ordered one Intel and one SolarFlare NICs ➢ With both I managed to drop anything that was below the 10G limit of the cards ➢ With SolarFlare I even tested uploading code into the NIC it self Data Plane Development Kit
- 21. Complex DPDKComplex DPDK ➢ Nobody, except me, was interested in supporting DPDK code ➢ Writing and updating DPDK is not trivial ➢ DPDK required specific HW that may not be available in the DataCenter Data Plane Development Kit
- 22. DPDK and P4DPDK and P4 ➢ A friend(Boyan Krosnov) told me about P4A friend(Boyan Krosnov) told me about P4 ➢ P4 made updating the logic and content of theP4 made updating the logic and content of the filter program a lot simpler for me...filter program a lot simpler for me... Data Plane Development Kit
- 23. P4 and peopleP4 and people ➢ P what?P what? ➢ If we were to use DPDK with P4, everyone hadIf we were to use DPDK with P4, everyone had to learn the language :(to learn the language :( Data Plane Development Kit
- 24. And then came XDPAnd then came XDPreceived packet XDP eBPF alloc_skb ingress (qdisc) bridge check broute brouting bridge check nat prerouting raw prerouting mangle forward nat prerouting bridging decision filter forward filter input routing decision filter forward mangle prerouting mangle forward filter forward mangle forward nat postrouting mangle postrouting mangle postrouting nat postrouting nat postrouting conntrack nat output filter output nat postrouting bdrige netfilter egress (qdisc) other net
- 25. And then came XDPAnd then came XDP ➢ Extremely fast and closest to the NIC, same asExtremely fast and closest to the NIC, same as DPDKDPDK ➢ Supported by many driversSupported by many drivers ➢ Extendable with eBPF functionsExtendable with eBPF functions ➢ Developed by Jasper BrouerDeveloped by Jasper Brouer eXpress Data Path
- 26. What I ended up, with?What I ended up, with? ➢ A filter similar to what CF did with their DROP example ➢ instead of a comparing a single prefix, I'm extracting the UDP data if the packet is UDP ➢ then the extracted data is compared with a BPF map ➢ I wrote a simple user space tool, that updates the map in the kernel ➢ voila I had a fast scrubber eXpress Data Path
- 27. the UDP scrubberthe UDP scrubber ➢ if the DNS request is not for a domain that is within the list in the map I drop the packet ➢ ToDo: add caching of responses with TTL eXpress Data Path
- 28. the TCP scrubberthe TCP scrubber This is where I had to stop :( ➢ compare the packet's dst port and allow it only if it is: ➢ SYN to a port that is allowed ➢ send and receive SYN cookies here ➢ part of already existing connection by examining its own db of tuples and the supplied by the user space(other VMs) eXpress Data Path
- 29. the TCP scrubberthe TCP scrubber This is where I had to stop :( ➢ It should handle the SYN cookie for the servers behind and replay the initial SYN if correct SYN,ACK is received eXpress Data Path
- 30. Testing the bastardTesting the bastard I knew I was able to drop packets fast... But I needed a proof ;) ➢ I had a talk with Jasper at Linux Plumbers 2019I had a talk with Jasper at Linux Plumbers 2019 ➢ He pointed me toHe pointed me to his patched version of pktgenhis patched version of pktgen on GitHub :)on GitHub :)
- 31. Now...Now... How to get from 10GbpsHow to get from 10Gbps to 200Gbps?to 200Gbps?
- 32. Now...Now... How to get from 10GbpsHow to get from 10Gbps to 200Gbps?to 200Gbps? ➢ Combining multiple VMs with ECMP ➢ I did that directly on the switch :)
- 33. LinksLinks How to drop 10 million packets per secondHow to drop 10 million packets per second https://blog.cloudflare.com/how-to-drop-10-million-packets/ https://github.com/cloudflare/cloudflare-blog/tree/master/2018-0 7-dropping-packets XDP tutorialXDP tutorial https://github.com/xdp-project/xdp-tutorial More XDP materials:More XDP materials: https://www.iovisor.org/technology/xdp Enhanced pktgen by JasperEnhanced pktgen by Jasper https://github.com/netoptimizer/network-testing
- 34. LinksLinks Linux tc and eBPFLinux tc and eBPF https://archive.fosdem.org/2016/schedule/event/ebpf/attachments/s lides/1159/export/events/attachments/ebpf/slides/1159/ebpf.pdf man pagesman pages http://man7.org/linux/man-pages/man8/tc-bpf.8.html http://man7.org/linux/man-pages/man2/bpf.2.html SolarFlare AOR firmware development kitSolarFlare AOR firmware development kit https://www.colfaxdirect.com/store/pc/viewPrd.asp?idproduct=1585 Data Plane Development KitData Plane Development Kit https://www.dpdk.org/ P4 Language SpecificationP4 Language Specification https://p4.org/p4-spec/docs/P4-16-v1.0.0-spec.pdf P4 meets DPDKP4 meets DPDK https://www.dpdk.org/wp-content/uploads/sites/35/2017/09/DP DK-Userspace2017-Day2-12-SANDOR_LAKI-T4P4S.pdf