Merging heterogeneous network measurement data
0 likes480 views
This document discusses merging heterogeneous network measurement data from different sources. It describes technologies like high-speed traffic measurements using 10Gbps network cards and Hadoop for processing large amounts of data. Alternative approaches for data integration like SQL databases and plain files are also covered. The challenges of processing logs from multiple sources at rates of 3M events per second are outlined. Hadoop and tools like Elasticsearch and Kibana are proposed for efficiently storing, accessing, and analyzing merged network and application measurement data.
Merging heterogeneous network measurement data
- 1. Use case: Merging heterogeneous network measurement data Jorge E. López de Vergara and Javier Aracil Jorge.lopez_vergara@uam.es Credits to Rubén García-Valcárcel, Iván González, Rafael Leira, Víctor Moreno, David Muelas, Javier Ramos, Paula Roquero, Carlos Vega and the rest of the HPCN-UAM team SMART Internet Monitoring Study 3rd Workshop, Barcelona, Spain, 22nd April 2015
- 2. Contents • Introduction • Technologies – High-speed traffic measurements – Data integration alternatives – Hadoop for network measurements – Log processing • Conclusions Merging heterogeneous network measurement data 2
- 3. Introduction • Network measurement data can come from different sources – Network-oriented sources: • SNMP MIB instances • Netflow records • Pcap files • … – Application-oriented sources: • Logs – Some standard (Apache web log) – Some proprietary (application specific) • Important to deal with encrypted traffic – It is necessary to provide ways to merge them Merging heterogeneous network measurement data 3
- 4. High-speed traffic measurements • Requirements – Capture at core networks – +10 Gbps links, sometimes virtualized – No packet drops • Available off-the-shelf resources that help on this tough task – Intel +10G network cards • Intel DPDK • Other developments: HPCAP at UAM – Mellanox +10G network cards • Mellanox Messaging Accelerator (VMA) – Multicore processors • CPU affinity and isolation for key tasks – Lots of RAM memory • Use of hugepages and mmap Merging heterogeneous network measurement data 4
- 5. High-speed traffic measurements Merging heterogeneous network measurement data 5 HPCAP M3 OMON Traffic Sniffer NIC Packet Ring Packet Buffer Packet Dumper Flow Manager Flow Exporter Packet-‐level Traces Flow-‐level Traces Agg.Stats Traces App. 1 App. 2 Flow Table READ (on demand) WRITE NON-‐VOLATILE MEMORY VOLATILE MEMORY KERNEL-‐LEVEL PROCESSING MODULE Packet Arrival USER-‐LEVEL PROCESSING MODULE App. N M3 OMON API packet access MRTG access flow access READ (real-‐time)
- 6. Data integration alternatives • SQL databases – Pros: reliable, normalized schemas, consistent with defined constraints – Cons: slow, need to use materialized views to go faster, creating materialized views is costly and sometimes it can’t be done concurrently à Not valid to deal with high-speed network measurements • Plain files, no SQL – Pros: much faster – Cons: inconsistencies, lack of normalization à Necessary to deal with high-speed network measurements, but keeping in mind its limitations Merging heterogeneous network measurement data 6
- 7. Hadoop for network measurements • General scenario Merging heterogeneous network measurement data 7 Flow Process Flows PCAP PC PC Users’ network Internet MapReduce Jobs HTTPSDNS Hive Hadoop Time series Popular web pages Malfunctioning Equipment Most used PCs Predictions SerDe HQL HDFS Admin Queries Raw traffic Processed traffic Deployed on many nodes to work properly Otherdata sources(logs)
- 8. Hadoop for network measurements • DNS analysis Merging heterogeneous network measurement data 8 DNS Traffic Whois database Intelligent Capture Module RADIUS records Hadoop Long-Term Storage NoSQL Database Data Visualization
- 9. Log processing • Requirements (real scenario): – Process 3M events per second (about 5 Gbps) • Put together application logs and network flows – Several disks in parallel are necessary to store the events at appropriate rate – Fast access to time series and aggregated statistics à What operators demand – Slower access to raw data à What IT analysts demand • Elasticsearch and Kibana tools provide some support, but it is necessary to tune them Merging heterogeneous network measurement data 9
- 10. Kibana UI Merging heterogeneous network measurement data 10
- 11. Conclusions • Need for different network measurement data sources – Combine network and application data – Necessary to find sources of problems • Is the slowdown caused by the network, the server or the application? • This question can only be answered if all information from different layers is provided and analyzed • Huge amount of data à it is necessary to work with fast processing systems • Availability of processing tools from the Cloud Computing community – It is necessary to adapt them to the network measurment processes Merging heterogeneous network measurement data 11