Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Kafka Summit 2020
0 likes1,312 views
The document discusses the implementation of data collection for a Security Information and Event Management (SIEM) system using Kafka Connect, focusing on the motivation, requirements, and anticipated challenges. It outlines two main connectors: 'push' and 'pull', as well as a transformations library to handle different message formats and improve data management. Key takeaways emphasize a highly available architecture, the importance of monitoring, and considerations for data archival and transformation.
Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Kafka Summit 2020
- 1. Feed Your SIEM Smart with Kafka Connect Vitalii Rudenskyi Security Architect, McKesson Corp
- 3. Why We Started Itâs not a mistake to make a mistake, but itâs a mistake to repeat the same mistake
- 4. Requirements â Logs collection to be vendor agnostic â Feed different analytics tools â Data filtering and cleaning â Existing data retention requirements â Scalable and highly available
- 5. Anticipated Challenges â Amount of data and number of sources â Variety of different formats â SaaS/PaaS applications and public Clouds â Cloud SIEM
- 7. The Three Keys That Open The Door
- 8. How to Collect Data
- 9. The Three Keys âPushâ Connector âPullâ Connector Transformations Library
- 10. Push - NettySource Connector â Unified: âall-in-oneâ different transport and protocols â Scalable: multiple tasks supported â Available: works behind a LB (health checks supported) â Configurable: multiple protocols (plain text, syslog, http, snmp, netflow) â Customizable: custom implementations supported
- 11. NettySource Connector Common deployment model
- 12. Pull - PollableAPIClient Connector â Simple: easy new connectors development â Scalable: multiple tasks/partitions supported â ConďŹgurable: interval and scheduled (cron-style) polls, retry/backoff, resettable offsets â Customizable: custom implementations supported
- 14. PollableAPIClient Connector 25+ ApiClient implementations
- 15. Transformations Library â Can transform different parts of a kafka message â Supports âifâ conditions
- 16. Takeaways
- 18. Headers All The Way â Track origin of the data â Conditional routing â Tagging in âchainedâ transformations â SIEM specific metadata
- 19. Syslog: Take Port 514 Under Control Multi-rules RegexRouter config
- 20. Data archiving solution â Pair of source and sink connectors â One connector for all topics â Compressed â Easy to restore
- 21. â Monitoring is essential â Keep original data â Be cautious when using âheavyâ transformations in source connectors
- 22. 5 kafka clusters 20+ kafka connect clusters 530+ deployed connectors 7+TB of data daily
- 23. What is the next...
- 24. We share! â NettySource Connector https://github.com/vrudenskyi/kafka-connect-netty-source â PollableAPIClient Connector https://github.com/vrudenskyi/kafka-connect-pollable-source https://github.com/vrudenskyi/kafka-connect-api-clients â Transformations Library https://github.com/vrudenskyi/kafka-connect-transform