HUGFR : Une infrastructure Kafka & Storm pour lutter contre les attaques DDoS en temps-réel par Steven Le Roux de la société OVH
4 likes5,111 views
The document discusses the architecture and functionality of an anti-DDoS solution employed by OVH, utilizing Kafka and Storm for real-time data processing and threat detection. It highlights the system's performance metrics, configurations, and challenges such as false positives and user behavior anomalies. Potential solutions for application-centric issues are also mentioned, along with resources for further learning.
HUGFR : Une infrastructure Kafka & Storm pour lutter contre les attaques DDoS en temps-réel par Steven Le Roux de la société OVH
- 1. AntiDDoS : Threat Detection Steven Le Roux Infrastructure Engineer
- 8. VAC
- 9. 9
- 14. 3 Tbps 17 Datacenters 32 PoPs
- 19. 19
- 23. Data Pipeline
- 25. Clients Producers Consumers Brokers Topics Partitions Replicas / kafka
- 26. / kafka / topic
- 27. / kafka / topic / replicas
- 28. / kafka / topic / replicas / factor / 3
- 29. / kafka / topics
- 30. / kafka
- 31. / kafka
- 32. / kafka / producers
- 35. Topology (DAG) Spouts Bolts Cluster Nimbus Supervisors Workers / storm
- 36. / storm / topology
- 37. / storm / topology / antiddos
- 39. Stream Grouping
- 40. / storm Shuffle Grouping Field Grouping Direct Grouping Other Grouping
- 41. Attacks Router Grouping Scans IP src Grouping / storm
- 42. Attacks ≈ 1s Scoring Filters Burst Scans IP Proto / storm
- 43. Indexing Prooving Producing / storm / event
- 44. #lifecycle
- 45. #dataviz
- 46. Nice speech… … so what ?
- 47. #issues False positives Strange behaviours from customers e.g. DB sync without connection pool Application centric i.e. UDP protocols
- 48. #solutions Add other sources Application Anti-DDoS Game Half Life/Source CS:GO TeamSpeak / Mumble GTA SA:MP … More to come (any special need ?)
- 50. #datalake
- 52. #hardware Nodes - Hardware CPU 16c/32t RAM 256GB Disks : OS : Raid 1 Data : 10 disks per node 200 MB/s ~ 1,5-2 Gbps
- 53. Kafka I/O bound Bench (1node) 1M+ msg/s No compression No ackers 80MB/s Tuning num.io.thread num.network.thread socket.*.buffer.* Storm #config CPU/RAM bound M+ tuples/s No ackers Break SRP Minimal workers Avoid transfer buffer
- 54. OpenSOC
- 55. Clément Sciascia - @csciasci Magnus Edenhill - @edenhillm https://github.com/edenhill/librdkafka LinkedIn - Apache Kafka Nathan Marz - Apache Storm #Thanks Storm basic training – Mickael G. Noll #more http://fr.slideshare.net/miguno/apache-storm-09-basic-training-verisign Kafka documentation
- 56. Steven LE ROUX @StevenLeRoux steven.le-roux@ovh.net Thanks