On the feasibility of 40 Gbps network data capture and retention with general purpose hardware
1 like170 views
This document discusses the design and implementation of a system for capturing and storing network data at 40 gbps using commercially available hardware. It outlines the motivations for traffic capture, the technical architecture of the system, and testing results which demonstrate its effectiveness in handling high-speed data. The conclusions highlight the feasibility of the approach, the ability to efficiently manage large data volumes, and proposed future enhancements for further optimization.
On the feasibility of 40 Gbps network data capture and retention with general purpose hardware
- 1. On the feasibility of 40 Gbps network data capture and retention with general purpose hardware SAC 2018 | Pau, France Guillermo Julián-Moreno Rafael Leira Jorge E. López de Vergara Francisco Gómez-Arribas Iván González April 10, 2018 Naudit HPCN & Escuela Politécnica Superior, Universidad Autónoma de Madrid
- 2. Outline 1. Introduction 2. Design and implementation 3. Results 4. Conclusions 1
- 3. Introduction
- 4. Motivation Why would we want to capture and store traffic? • Online analysis and monitoring (e.g., flow records, traffic volume dashboards, IDS). • Data retention for specialized analysis or policy requirements (e.g. GPDR). 10 GbE standard is widespread, now we are seeing higher speeds (40 GbE, but also 100 GbE) • 10 GbE is the “last” standard that we can process with a single core. • 40 GbE and higher speeds require parallelism: how? 2
- 5. Purpose of the system • Receive the traffic at 40 Gbps as efficiently as possible. • Timestamp the incoming traffic. • Store the network frames in the disk at 40 Gbps. • Use commercial off-the-shelf hardware to reduce costs. 3
- 7. Previous architecture DMA tail head Descriptor ring Data bufferNIC • Single thread copying frames to the intermediate buffer. • Write files by blocks and use padding at the end of each file. • Return the descriptor’s ownership to the card after the copy (no allocations needed). 4
- 8. Reading from NIC and copying to buffer 1 23 4 Rx Intermediate buffer ... • Usual approach for parallelism is RSS queues. Problem: ensuring uniform distribution between queues. • Given our limited scope, switch to single queue and fixed descriptor assignments: uniform distribution and no synchronization required for reading. • Use a single atomic counter for the buffer write offset: as fast as possible and no deadlocks possible. • Add padding to the beginning and end of the files to avoid having frames overrunning file boundaries. 5
- 9. Client reading Kernel Userspace Rx 0 Rx 1 Rx 2 Rx 3 1. Allocation 2. Copy Client … 3. New data! 4. Reading • Userspace clients get last written byte via syscalls and set their last read byte. • RX thread 0 updates s, the space available in the buffer. No thread writes more than ⌊s/n⌋ bytes in a batch. 6
- 10. Writing to disk Two options for the write process: • Regular files written in 4MB blocks. Needs a fast filesystem. • Distributed writes between several NVMe disks with SPDK. Features to reduce hardware requirements: • Simple filtering system that searches “strings” of bytes at fixed positions. • Selective storage: only store the first N bytes of each frame. 7
- 11. Results
- 12. Hardware used Traffic generator RX Server 1 RX Server 2 CPU Intel Xeon E5-1620 v2 Intel Xeon E5-1620 v2 2 × Intel Xeon E5-2630 v4 Clock 3.70GHz 3.70GHz 2.20 GHz Cores 4 4 2 x 10 Memory 32 GB 32 GB 2 x 64 GB NIC Intel XL710 Intel XL710 Intel XL710 Storage SATA RAID SATA RAID 6 x NVMe Est. cost 7,000e 7,000e 10,000e Table 1: Specifications of the servers used for testing. HyperThreading was disabled. 8
- 13. Storage speed 0 10 20 30 40 50 2 3 4 5 6 Rate(Gbps) Number of discs Disk write rate Software RAID SPDK Theoretical max speed Figure 1: Performance of the NVMe disk array. 9
- 14. Traffic generation 0 5 10 15 20 25 30 35 40 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 Rate(Gbps) Frame size (bytes) Generation rate Send rate Theoretical max rate Figure 2: Synthetic traffic rates achieved with our custom, DPDK-based traffic generator. We also made a version capable of sending large PCAP files at line rate. 10
- 15. Timestamping accuracy Frames Mean Std All 1738 ns 3296 ns One out of every eight 55 ns 287 ns Table 2: Timestamping accuracy. The Intel NIC posts descriptors in batches of eight, so we have to take that into account for the accuracy. 11
- 16. Traffic capture 0 25 50 75 100 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 0 5 10 15 20 25 30 35 40 Loss% Rate(Gbps) Frame size (bytes) Capture rate Lost % Port Drop % Send rate Capture rate Theoretical max rate Figure 3: Results of the first test: retrieval of the frames from the NIC. The bottleneck is the card for small frame sizes. 0 25 50 75 100 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 0 5 10 15 20 25 30 35 40 Loss% Rate(Gbps) Frame size (bytes) Capture rate Lost % Port Drop % Send rate Capture rate Theoretical max rate Figure 4: Results of the second test: writing of the frames to a null device. 12
- 17. Traffic storage 0 25 50 75 100 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 0 5 10 15 20 25 30 35 40 Loss% Rate(Gbps) Frame size (bytes) Capture rate Lost % Send rate Capture rate Theoretical max rate Figure 5: Results of the third test: traffic storage using SPDK. 13
- 18. Traffic storage Name Size Avg. frame size Send rate Loss % CAIDA 222 GB 787.91 B 39.78 Gbps <0.01 University 4.3 GB 910.08 B 39.82 Gbps 0 Table 3: Performance on reception of traffic capture files. 14
- 19. Conclusions
- 20. Results • We have created and open-sourced a system capable of capturing, timestamping and storing network traffic at 40 Gbps. • Not using RSS parallelism is feasible and useful in our limited-scope system. • The one-copy mechanism and synchronization algorithms allow our system to store line-rate traffic at frame sizes of 300 bytes and above (enough for the majority of environments). • We have created a testbed capable of saturating 40 GbE links for frames of size 96 bytes or greater. 15
- 21. Future work • Improve the selective-storage approach: more effective filters (ASCII/BPF) or limits based on RX rate. • Detailed comparison of the disorder and timestamp inaccuracies between our approach and RSS queues. • Port this system to virtual machines with SR-IOV NVFs. 16
- 22. Questions?