FastNetMon and Metrics
0 likes327 views
FastNetMon is a comprehensive DDoS detection and network traffic management tool offering capabilities for flow analytics on platforms like AWS and Google Cloud. The document discusses various metrics storage options such as Graphite, InfluxDB, and ClickHouse, comparing their performance, scalability, and usability. Additionally, it provides contact information for the author, Pavel Odintsov, and details about the FastNetMon community.
FastNetMon and Metrics
- 2. Hello 2 I’m Pavel Odintsov, the author of FastNetMon: https://fastnetmon.com Ways to contact me: ● linkedin.com/in/podintsov ● github.com/pavel-odintsov ● twitter.com/odintsov_pavel ● IRC, Libera Chat, pavel_odintsov ● pavel@fastnetmon.com
- 3. What is FastNetMon 3 ● DDoS detection Tool ● Amazon AWS VPC Flow Analytics ● Google Compute VPC Flow Analytics ● BGP Blackhole Automation Tool ● BGP Automation Tool ● Traffic Visibility Tool ● Network Automation Tool ● Network Traffic Engineering Tool
- 6. FastNetMon: Detection Logic 6 Detection type: • Threshold based (based on host’s average traffic) THRESHOLD TYPES: • USING TOTAL TRAFFIC • USING TOTAL PPS RATE • PER PROTOCOL
- 7. FastNetMon: Tra ic Reports in Grafana 7
- 8. FastNetMon: More Tra ic Reports in Grafana 8
- 10. Our Challenges with Metrics 10 ● Cardinality: medium sized network has ~1m IPv4 hosts with 32 metrics each ● Value range from 0 to UINT64_MAX: traffic in mbit / s or packet /s ● 1s precision ● Insert in very large batches ● Customers love top-k queries
- 11. Graphite As Storage for Metrics, 2015 11 ● “.” as delimiter. IPs and prefixes look ugly: 10_1_2_3 , 10_1_2_3_24 ● Limited by performance of single CPU core ● Disk space hungry datastore format ● Whisper is simple and easy to implement ● Graphite is not well maintained and broken in recent Debian / Ubuntu
- 12. Graphite Web: World Before Grafana 12
- 15. InfluxDB As Storage for Metrics, 2016 15 ● Allows “.” in metric names ● Pretty compact datastore format ● Automated retention ● Native Grafana support ● Can use multiple CPU cores ● Easy installation, just single binary ● May need tens of minutes for loading with large database ● Uses lots of memory ● Top-k query is extremely slow ● Does not scale after 2m metrics per second ● Queries over few days of data are very slow
- 16. ClickHouse As Storage for Metrics, 2022 16 ● Allows “.” in metric names ● Pretty compact datastore format ● Automated retention ● Plugin for Grafana ● Can use multiple CPU cores ● Requires SSE 4.2 :( ● Top-k query is pretty fast ● Supports unlimited cardinality ● Queries over few days of data can be finished in reasonable time ● Ability to store flows!
- 17. Clickhouse vs InfluxDB 17 InfluxDB ClickHouse Cardinality < 1m of unique series Battle tested with 16m+ unique series Metrics / s < 1m per second 10m+ per second Top-k performance Extremely slow Good Data format Inefficient, text Very Efficient, binary Query syntax Counterintuitive Well known SQL Multi CPU support Limited Brilliant, scales linearly Grafana Native Plugin based
- 19. ClickHouse as Flow Storage 19
- 21. Can We Retrieve Flows? 21
- 22. Can We Show Flows? 22
- 23. FastNetMon: our community 23 ● Site: https://fastnetmon.com ● GitHub: https://github.com/pavel-odintsov/fastnetmon ● IRC: #fastnetmon at Libra Chat ● Telegram: https://t.me/fastnetmon ● Slack: http://bit.ly/2o5Idx8 ● LinkedIN: https://www.linkedin.com/company/fastnetmon/ ● Facebook: https://www.facebook.com/fastnetmon/ ● WhatsApp: https://chat.whatsapp.com/JjwF855pwZvIIasTUsZ7EO
- 24. THANKS! ANY QUESTIONS? You can find me at: ⬥ @odintsov_pavel ⬥ pavel@fastnetmon.com ⬥ linkedin.com/in/podintsov 24