Cloud-Scale BGP and NetFlow Analysis
0 likes550 views
The document discusses the challenges and solutions for cloud-scale BGP and netflow analysis, highlighting the need for accurate and timely network operations visibility. It emphasizes the importance of utilizing multiple data sources for comprehensive insights and addresses the limitations of existing tools in handling big data from network monitoring. Kentik's SaaS solution is presented as an effective approach to achieve real-time analytics and management at terabit scale.
Cloud-Scale BGP and NetFlow Analysis
- 1. Cloud-Scale BGP and NetFlow Analysis Jim Frey, VP Product, Kentik Technologies December 15, 2015
- 2. 2 • Common NetOps Stress points • Helpful Data Sets – NetFlow, BGP • Handling NetFlow and BGP at Cloud Scale • Kentik’s Approach • Wrap-Up / Q&A Agenda
- 3. R R S S S S S R R S S S S S NetOps Stress Points: Needing Instant Answers How should I allocate my resources in the future? Does performance meet expectations? Is this an attack or legitimate traffic? Where in my network is the problem? Things You Need Answers to About/From Your Network $$$ $$$ $$$ X
- 4. 4 • Accurate Visibility, Without Delay • Relevant Alerts: No False Positives or Negatives • Complete Data: Breadth + Depth • Fast/Flexible Data Exploration • Tools that don’t suck (time or $$) What We Hear…. To Address These Questions, NetOps Needs:
- 5. 5 What Data Sets Can Help? And which ones can do the job cost effectively?
- 6. 6 Primary Network Monitoring Data Choices Examples - SNMP, WMI Advantages - Ubiquitous - Good for monitoringdevice health/status/activity Disadvantages - Notraffic detail - Typically nofrequentthan every 5 minutes truly anti- real-time Polled Stats Examples - NetFlow, sFlow, IPFIX Advantages - Details on traffic src/dest/content, etc. - Very costeffective Disadvantages - NRT(near real-time)atbest - Incomplete app-layer detail - Limitedperformance metrics - Data volumes can be massive Flow Records Examples - Packets -> xFlow - Long term stream-to-disk Advantages - Mostcomplete app layer detail - True real-time (millisecondlvl) - Complete vendor independent Disadvantages - Expensive todeploy at scale - Requires network tapor SPAN - Packetcaptures can be massive Packet Inspection
- 7. 7 Secondary Network Monitoring Data Choices Examples - Syslog Advantages - Continuous/streaming - Unique, device-specific info - True real-time Disadvantages - Nostandards – musthave very flexible search/mappingtools - Data volumes can be massive Log Records Examples - OSPF, IGRP, BGP Advantages - Details on traffic paths and provider volumes - Insights intoInternetfactors Disadvantages - Address data only – no awareness of traffic - Mustpeer with routers to get updates Routing/Path Data Examples - IP SLA, Independenttestsw Advantages - Assess functions/services 24x7 - Provides both availability and performance measures Disadvantages - Deploying/maintainingenough agents to achieve full coverage - Only an approximation of real user experience (atbest) Synthetic Agents
- 8. 8 • You never know which data set will present the specific insights you need • The challenge (real magic) comes from correlating multiple datasets, i.e.: • Behavioral observations with configuration changes • Trends with underlying traffic details • Routing data with traffic data Key Assertion: Use Multiple Data Types for Best Results
- 9. 9 For Providers • Recognizing newservice opportunities basedon subscriber(and peer) behavior • Optimizing peering relationships forcostcontrol For Web Services/ Commerce • Recognizing where yourcustomers are andhowtheyreach you • Managing peering relationships forbestcustomerexperience For Enterprise • Assessing howyourconnectivityproviders perform/compare • Building InternetIQ – howyou connect/relate to the outside world Why Correlate Routing Data with Traffic Data?
- 10. 10 Cloud Scale for NetFlow and BGP: The Big Data Challenge Why can’t we just use our existing tools?
- 11. Cloud, SaaS, Big Data Network traffic has grown exponentially; Legacy tools/tech haven’t kept pace. Result? Fragmented tools, visibility gaps, unanswered questions. Existing Tools: Falling Behind 10M 100M 1G 10G 100G
- 12. 12 - Network Monitoring Data IS Big Data - Meets Volume/Variety/Velocity Test - Billions of records/day (millions/second) - Big Data architectures are considered best practices today for open/flexible correlation, analytics Why Big Data?
- 13. 13 Existing solutions shortfalls: - Flexibility for moving between viewpoints and into full details - Data Completeness due to reliance on summarized/aggregated flow data - Speed: Generating new analysis in a timely manner Specific Challenges For NetFlow + BGP - Network Monitoring Data IS Big Data - Meets Volume/Variety/Velocity Test - Billions of records/day (millions/second) - Big Data architectures are considered best practices today for open/flexible correlation, analytics Why Big Data?
- 14. 14 How to Get/Use Big Data Approach?
- 15. 15 1. BYO – Build Your Own • Pick back end & reporting/analysis tools (open source = free?) • Procure operating platforms (hard, virtual, or cloud servers = $$) • Integrate, add data sources, and get it up and running (dev = $$) • Keep it up and running (ops/admin = $$) How to Get/Use Big Data Approach?
- 16. 16 1. BYO – Build Your Own • Pick back end & reporting/analysis tools (open source = free?) • Procure operating platforms (hard, virtual, or cloud servers = $$) • Integrate, add data sources, and get it up and running (dev = $$) • Keep it up and running (ops/admin = $$) 2. Let SOMEONE ELSE build/optimize/operate • Subscribe to SaaS (ops $$) • Just Send Your Data and enjoy the ride! How to Get/Use Big Data Approach?
- 17. 17 Kentik’s Answer How we address the Big Data challenge to meet the needs of Network Operators now
- 18. Kentik Detect: the first and only SaaS Solution For Network Ops Management & Visibility at Terabit Scale CL OU D -B A S E D RE A L -TIM E M U LTI-TE N A N T OP E N G L OB A L Analyze & Take Action Big Data Network Telemetry Platform S S S R R The Network is the Sensor Web Portal Real-time & historical queries NetFlow/ sFlow/IPFIX SNMP BGP Alerts E-mail / Syslog / JSON Open API SQL / RESTful Kentik Data Engine
- 19. Multi-tiered/Clustered for Scale / Load Balancing / HA, Hosted by Kentik What’s Behind the Kentik Data Engine POSTGRES SERVERS SQL DATA STORAGE CLUSTER NetFlow SNMP BGP INGEST CLUSTER CLIENTS N M Optimized forMassive DataIngest & Rapid Query Response
- 22. 22 Traffic by Source Geography
- 24. 24 AS Top Talkers and Drill Down Options
- 25. 25 Peering Analytics: ASN by Dest Country Paths
- 26. 26 Peering Analytics: Traffic by BGP Paths
- 27. 27 Peering Analytics: Traffic by Origin AS (“Last Hop”)
- 28. 28 Peering Analytics: Traffic by Transit AS
- 29. Key Takeaways: Cloud Scale NetFlow + BGP Why You Need It - Clear Insight into external/Internet network traffic behaviors - Improved customer/subscriber engagement - Reduced network operating costs Technical Path to Success - This is a big data problem, requiring high capacity/speed for data management, correlation, exploration, and analytics - SaaS solutions are a fully viable option
- 30. Network Intelligence at Terabit Scale Thank You! Jim Frey VP Product KentikTechnologies jfrey@kentik.com @jfrey80