Kernel Recipes 2015: Kernel packet capture technologies
5 likes2,874 views
The document discusses kernel packet capture technologies and their evolution, focusing on methods such as raw sockets, libpcap, af_packet, pf_ring, and netmap. It highlights the significance of network intrusion detection systems like Suricata and challenges faced in high-speed traffic inspection. The author emphasizes the need for advancements in packet capture architecture to handle increasing network speeds efficiently.
!["The End of the Internet"
[raw socket ...] spells catastrophe for the integrity of the Internet.
Steve Gibson in 2001
Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 7 / 54](https://image.slidesharecdn.com/2015kernelrecipescapture-151005202119-lva1-app6891/85/Kernel-Recipes-2015-Kernel-packet-capture-technologies-5-320.jpg)
!["The End of the Internet"
Talking about introduction of raw socket in MS Windows
Allow users to write any packets
Could be used to abuse protocol and [poorly implemented] OS
More info at http://www.informit.com/articles/article.aspx?p=27289
Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 8 / 54](https://image.slidesharecdn.com/2015kernelrecipescapture-151005202119-lva1-app6891/85/Kernel-Recipes-2015-Kernel-packet-capture-technologies-6-320.jpg)
Kernel Recipes 2015: Kernel packet capture technologies
- 1. Kernel packet capture technologies Ăric Leblond Stamus Networks October 1, 2015 Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 1 / 54
- 2. 1 Introduction 2 Why capture 3 Libcap and raw socket 4 AF_PACKET 5 PF_RING 6 AF_PACKET goes multi* 7 Netmap 8 Latest AF_PACKET evolution 9 ++zero copy 10 Conclusion Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 2 / 54
- 3. Ăric Leblond Co-founder of Stamus Networks Company providing network probe based on Suricata Focusing on bringing you the best of Suricata IDS technology Open source hacker Suricata core developer NetïŹlter core team member Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 4 / 54
- 4. Raw socket: deïŹnition A raw socket is an internet socket that allows direct sending and receiving of Internet Protocol packets without any protocol-speciïŹc transport layer formatting. Wikipedia Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 6 / 54
- 5. "The End of the Internet" [raw socket ...] spells catastrophe for the integrity of the Internet. Steve Gibson in 2001 Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 7 / 54
- 6. "The End of the Internet" Talking about introduction of raw socket in MS Windows Allow users to write any packets Could be used to abuse protocol and [poorly implemented] OS More info at http://www.informit.com/articles/article.aspx?p=27289 Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 8 / 54
- 7. Raw socket: usage Send and receive Send low level message: icmp, igmp Implement new protocol in userspace SnifïŹng Capture trafïŹc Promiscuous mode Use by network monitoring tools Debugging tools: tcpdump, wireshark Monitoring tools: iptraf, ntop, NSA Intrusion detection systems: snort, bro, suricata Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 9 / 54
- 8. Network Intrusion Detection System: deïŹnition An intrusion detection system (IDS) is a device or software application that monitors network or system activities for malicious activities or policy violations and produces reports to a management station. Wikipedia Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 10 / 54
- 9. Network Intrusion Detection System: challenge IDS detection rule Some data Complexity of rule Work on recontructed stream Protocol ïŹeld analysis Pattern recognition on ungzipped content (http_server_body) Got around 15000 rules in standard ruleset Need to inspect 10Gbps of traïŹc or more Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 11 / 54
- 10. Suricata: Open source & multi threaded IDS IDS and IPS engine Get it here: http://www.suricata-ids.org Project started in 2008 Open Source (GPLv2) Funded by consortium members (and originaly US government) Run by Open Information Security Foundation (OISF) More information about OISF at http://www.oisf.net/ Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 12 / 54
- 11. Suricata Features High performance, scalable through multi threading Protocol identiïŹcation File identiïŹcation, extraction, on the ïŹy MD5 calculation TLS handshake analysis, detect/prevent things like Diginotar Hardware acceleration support: Useful logging like HTTP request log, TLS certiïŹcate log, DNS logging Lua scripting for detection Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 13 / 54
- 12. libpcap Multi OS abstraction for packet capture All *nix, Windows Multi layer: Network, USB, . . . Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 15 / 54
- 13. Raw socket: the initial implementation A dedicated socket type #include <sys / socket . h> #include < netinet / in . h> raw_socket = socket (AF_INET , SOCK_RAW, int protocol ) ; Straight socket mode Get packet per packet via recvmsg Optional ioctl Get timestamp Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 16 / 54
- 14. Memories of another time "640 K ought to be enough for anybody." Memory contraint design No preallocation On demand only Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 17 / 54
- 15. Disclaimer Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 18 / 54
- 16. IDS design Monoprocess No Performance for you, go home now. Marty Roesch about multithread and network data processing, 2010 Suricata architecture Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 19 / 54
- 17. NAPI (2001-200?) Reducing interrupts usage Interrupts tempest at high packet rate All CPU time is sued to handle the interrupts NIC driver needs to be updated No direct change for packet capture Change internal to device driver Direct performance impact on packet capture Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 21 / 54
- 18. NAPI performance Table extracted from luca.ntop.org/Ring.pdf Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 22 / 54
- 19. Problem of the socket mode Internal path Data in card buffer Data copied to skb Data copied to socket Data read and copied by userspace Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 23 / 54
- 20. Memory map approach Sharing is the solution Kernel expose some memory Userspace access memory directly Spare a message sending for every packets mmap internal path Data in card buffer Data copied to skb Data copied to ring buffer Userspace access data via pointer in ring buffer Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 24 / 54
- 21. TPACKET_V2 setup socket(): creation of the capture socket setsockopt(): allocation of the circular buffer (ring) via PACKET_RX_RING option mmap(): mapping of the allocated buffer to the user process capture poll(): to wait for incoming packets shutdown close(): destruction of the capture socket and deallocation of all associated resources. Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 25 / 54
- 22. Memory organization Ascii art block #1 block #2 +---------+---------+ +---------+---------+ | frame 1 | frame 2 | | frame 3 | frame 4 | +---------+---------+ +---------+---------+ block #3 block #4 +---------+---------+ +---------+---------+ | frame 5 | frame 6 | | frame 7 | frame 8 | +---------+---------+ +---------+---------+ Components Frame contains a datagram data Blocks are physically contiguous region of memory Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 26 / 54
- 23. Performance Graph extracted from luca.ntop.org/Ring.pdf Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 27 / 54
- 24. Suricata architecture MMAP option Support of TPACKET_V2 Zero copy mode Implied changes Access data via pointer to ring buffer cell Release data callback Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 28 / 54
- 25. PF_RING original design (2004) Architecture ring design mmap capture only interface skip kernel path put in ring buffer and discard user access the ring buffer Project Project started by Luca Deri Available as separate sources Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 30 / 54
- 26. PF_RING performance Show real improvement on small size packets Pre optimisation result Better result in following version due to a better poll handling Table extracted from luca.ntop.org/Ring.pdf Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 31 / 54
- 27. PF_RING going multicore (around 2008?) Sharing the load Each core has a ïŹnite bandwidth capability Multicore CPU were introduced in 2006 Sharing load become common Previously separate hardware was used to split the network load Straight forward solution Allow multiple sockets to be attached to one interface Load balance over the attached sockets Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 32 / 54
- 28. Suricata autofp multi reader Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 33 / 54
- 29. PF_RING code Build system and sources Custom build system No autotools or cmake Include patched drivers SVN stats g i t log ââformat=format : "%s " | sort | uniq âc | sort ân | t a i l ân10 15 Minor change 20 f i x 20 minor changes 22 l i b refresh 30 Library refresh 43 minor change 67 minor f i x Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 34 / 54
- 30. David Miller in da place Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 36 / 54
- 31. AF_PACKET load balancing (2011) Multiple sockets on same interface Kernel does load balancing Multiple algorithms LB algorithm Round-robin Flow: all packets of a given ïŹow are send to the same socket CPU: all packets treated in kernel by a CPU are send to the same socket Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 37 / 54
- 32. AF_PACKET CPU Load balancing RSS queues Multiqueue NIC have multiple TX RX Data can be split in multiple queues Programmed by user Flow load balanced RSS queues load balancing NIC does load balancing using hash function CPU afïŹnity is set to ensure we keep the cache line Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 38 / 54
- 33. Suricata workers mode Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 39 / 54
- 34. tpacket_v3 (2011) The problem Cell are ïŹxed size Size is the one of biggest packet (MTU) Small packets use same memory as big one Variable size cells Ring buffer Update memory mapping to enable variable sizes Use a get pointer to next cell approach Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 40 / 54
- 35. Netmap (2012) Similar approach than PF_RING skip kernel path put in ring buffer and discard User access the ring buffer Paired with network card ring More info http://queue.acm.org/detail.cfm?id=2103536 Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 42 / 54
- 36. Performances Table by Luigi Rizzo Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 43 / 54
- 37. AF_PACKET rollover option (2013) Single intensive ïŹow Load balancing is ïŹow based One intensive ïŹow saturate core capacity Load needs to be shared Principle Move to next ring when ring is full As a load balancing mode As a fallback method Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 45 / 54
- 38. Rollover and suricata (1/2) Graph by Victor Julien Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 46 / 54
- 39. Rollover and suricata (2/2) A TCP streaming issue Rollover activation lead to out of order packets Fool TCP stream reconstruction by suricata Result in invalid streams Possible solution Evolve autofop multicapture Decode and dispatch packets Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 47 / 54
- 40. DPDK (2012-) Data Plane Development Kit set of libraries and driver design for fast packet processing impact on software architecture Architecture multicore framework huge page memory ring buffers poll-mode drivers Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 48 / 54
- 41. Suricata workers mode limit Packet treatment can be really long Involve I/O on disk or network Huge computation like regular expression Ring buffers are limited in size A slow packet can block a whole buffer Suricata need to dequeue faster Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 50 / 54
- 42. Need to evolve Suricata architecture Switch to asynchronous Release ring buffer elements as fast as possible Buffer in userspace An enhanced autofp approach? Fast decode Copy data to packet pool of detect thread With a fast decision Release data Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 51 / 54
- 43. Conclusion (1/2) A small subject and a huge evolution Has follow evolution of hardware architecture Always need to deal with more speed 10Gbps is common 100Gbps is in sight Multiple technologies Vanilla kernel propose some solutions Patching may be required to do more Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 53 / 54
- 44. Conclusion (2/2) Do you have questions ? Contact me Mail: eleblond@stamus-networks.com Twitter: @Regiteric More information Suricata: http://www.suricata-ids.org PF_RING: http://www.ntop.org/products/packet-capture/pf_ring/ netmap: http://info.iet.unipi.it/~luigi/netmap/ dpdk: http://dpdk.org/ Ăric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 54 / 54