Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Traffic Analysis
1 like110 views
The document discusses a proposed method for real-time sub-flow traffic analysis using a 'packet traffic genome' concept, which aims to optimize virtual machine (VM) placement and migration in data centers. It emphasizes the use of patterns derived from packet data and suggests transitioning to a Bayesian classifier for improved traffic state classification. The document highlights the significance of effective packet capture and the relationship between traffic patterns and data classification.
Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Traffic Analysis
- 1. Marat Zhanikeev maratishe@gmail.com maratishe.github.io Sub-Flow Traffic Analysis Tokyo Univ. of Science Packet Traffic Genome Project NETSAP/COMPSACW 2018 @NII/Tokyo PDF → bit.do/180727 #packets #traffic #subflow #GA #genome #datacenter Towards a as a Method for Realtime
- 2. (several) General Observations • in actual genome research, the concept of junk DNA is firmly established ?? • ... but has anyone heard of a GA algorithm with gene memory? • related example: when optimizing VM placement in DCs, would you ① bin-pack all VMs at regular intervals or rather ② formulate a migration-centric cost-aware problem 01 ◦ hint: full repacking is not feasible as it causes massive VM placement jitter • proposal: let’s look into GA with memory concept for traffic processing ◦ note: genome and GA accidentally overlap, but GA is not a strong requirement (any optimization should do) • closest rivals: the ”let’s look at traffic at subflow/flowlet/burst/train level” crowd 03 01 myself+0 ”Optimizing Virtual Machine Migration for Energy-Efficient Clouds” IEICE Trans. Comm (2014) 03 K.He+5 ”Presto: Edge-based Load Balancing for Fast Datacenter Networks” ACM SIGCOMM Com.Comm Review (2015) Marat Zhanikeev – maratishe@gmail.com Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Traffic Analysis – bit.do/180727 2/13 2/13
- 3. First, Capture Packets Effectively • between advanced capture drivers and massively multicore 02 processors, not a problem even in modern DCs (10Gbps and above) Global Networks Data Center Internals Gateway Switch Capture Manager CPU CPU CPU CPU CPU CPU … Storage Mirror 02 myself+0 ”A lock-free shared memory design for high-throughput multicore packet traffic capture” IJNM (2014) Marat Zhanikeev – maratishe@gmail.com Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Traffic Analysis – bit.do/180727 3/13 3/13
- 4. Raw Packets to Genome • using 2-3 intervals on the log scale, cut flows into bursts/trains and classify them using ① packet count and ② interval (gap) • two consecutive lower letter patterns (lazy keep-alive?) are not recorded, and are used to cut genome into words Increasing packet gap #ofpacketsinaburst Captured packets (per flow) F A B C a b c aBcF aFaFbF Marat Zhanikeev – maratishe@gmail.com Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Traffic Analysis – bit.do/180727 4/13 4/13
- 5. Looking for Patterns in Genome Strings 0 1 2 3 4 5 log( 1 + gap) in microseconds 0 2 4 6 8 10 12 14 spagevitucesnocfo# F A a b c a A b B c • meaningful sequences representing valid dynamics in traffic – see C1...C6 0 1 2 3 4 5 log( 1 + gap) 0 2 4 6 8 10 12 14 16 No.consecutivegaps C1 C2 C3 C4 C5 C6 Marat Zhanikeev – maratishe@gmail.com Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Traffic Analysis – bit.do/180727 5/13 5/13
- 6. Yet More Patterns • few As, various Fs, and other patters are obvious F A B C a b c 0 0.2 0.4 0.6 0.8 1 Countdistribution F A B C a b c 0 0.2 0.4 0.6 0.8 1 Countdistribution wand wide Marat Zhanikeev – maratishe@gmail.com Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Traffic Analysis – bit.do/180727 6/13 6/13
- 7. Meaningful Genome Sequences • again, many valid sequences but ① small overall ratio (feature, not bug) and ② long words are increasingly rare 0 5 10 15 20 25 30 35 40 45 Gene length 0 0.01 0.02 0.03 Occurenceprobability wand wide Marat Zhanikeev – maratishe@gmail.com Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Traffic Analysis – bit.do/180727 7/13 7/13
- 8. Genome → Traffic State Classification 0 100 200 300 400 500 Time sequence 0 1 2 3 4 5 6 7 Clusteraffiliation Cluster counts : 66 291 20 57 33 23 10 0 100 200 300 400 500 Time sequence 0 1 2 3 4 5 6 7 Clusteraffiliation Cluster counts : 73 316 68 14 13 6 10 wand wide • simple kmeans Marat Zhanikeev – maratishe@gmail.com Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Traffic Analysis – bit.do/180727 8/13 8/13
- 9. Genome → Traffic State Classification (2) • the same kmeans data, showing that there is overall majority of states are of the same type (idle) • the 80% of the same kind of state is perfect for the junk DNA element in optimizations 0 40 80 120 160 200 240 Decreasing sequence 0.2 0.6 1 1.4 1.8 2.2 Occurencecountaslog(1+y) wand wide Marat Zhanikeev – maratishe@gmail.com Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Traffic Analysis – bit.do/180727 9/13 9/13
- 10. Wrapup and Future Work • even with plain kmeans, a drastically uneven distribution of traffic states (judging from genome sequences) is discovered – perfect for the junk DNA element • next step: migrate to Bayesian classifier as a better technique ◦ HMI: humans identify and define patterns, machines remember them by incorporating the patterns as junk DNA (at runtime) Marat Zhanikeev – maratishe@gmail.com Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Traffic Analysis – bit.do/180727 10/13 10/13
- 11. That’s all, thank you ... Marat Zhanikeev – maratishe@gmail.com Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Traffic Analysis – bit.do/180727 11/13 11/13
- 12. Advanced Packet Capture PF_RINGs meter User space Kernel Capture thread Capture thread Capture thread NIC Driver … Manager Network Network To/from Collector • PF_RING is one of the best (kernel level) methods for streamlining packet capture • then, it is a matter of efficiency splitting the job across multiple cores 02 • the concept of massively multicore processors is applicable as well 02 myself+0 ”A lock-free shared memory design for high-throughput multicore packet traffic capture” IJNM (2014) Marat Zhanikeev – maratishe@gmail.com Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Traffic Analysis – bit.do/180727 12/13 12/13
- 13. Trace-based Analysis • real packet traces are preferred to simulated traffic • WAND 04 and WIDE 05 traffic archives are among the best known in this area, this paper used both for comparison 04 ”WAND: Waikato Internet Traffic Storage” https://wand.net.nz/wits (current) 05 ”MAWI Working Group Traffic Archive: Packet Traces from WIDE Backbone” http://mawi.wide.ad.jp/mawi (current) Marat Zhanikeev – maratishe@gmail.com Towards a Packet Traffic Genome Project as a Method for Realtime Sub-Flow Traffic Analysis – bit.do/180727 13/13 13/13