Apache kafka
- 2. About.me Desenvolvedor de software +10 anos Ilegra desde novembro / 2016 Arquitetura Unicred Projeto core banking Estudado: Microservices Reactive services
- 3. About History Motivation What is? Design concepts Use case
- 4. History
- 5. Motivation Complex infrastructure Different tools Performance issues Governance Persistent
- 6. Motivation
- 7. Motivation
- 8. Motivation
- 9. Motivation
- 10. What is?
- 11. What is?
- 12. What is? Distributed streaming platform You can produce and consume messages similar to conventional message systems. Store data - fault tolerant Process events in order
- 14. Log Persistent sequence messages Newest messages are append to the log Similar to database transaction log State recovery Sequence of actions / events Debug Audit
- 15. Topic Logical division Subject Unit of coupling Can be split into partitions
- 16. Partition
- 17. Partition Fisic unit Each partition are a log Producers and consumers connect Split into brokers
- 18. Partition
- 19. Producer Write messages to partitions Round Robbin - Balanced Semantic Strategy - Hot Spot attention Messages can be compressed Sync Send and wait for response. Exception will be treat and resend manually Async Register a callback
- 20. Producer Several configs acks = 0 ■ Very fast. ■ Without consistent guarantee acks = 1 ■ Wait the response of the leader ■ Sync can increase the latency acks = all ■ Wait for all replication process
- 21. Consumer Consume messages Control the offset Commit the offset Have a consumer group
- 22. Consumer
- 23. Consumer
- 24. Consumer
- 25. Consumer
- 26. Consumer
- 27. Summary Topic can have partitions Each partition is a log Producers send messages Consumers pull messages and control the offset Each consumer group receive just one message per partition Number of partitions is the unit of parallel processing
- 28. Use to connect data systems to kafka Can override common ETL systems Available connectors: Amazon S3 JDBC, MYSQL HANA Cassandra Elastic Search FTP Connector
- 29. Who are using?
- 30. Who are using? ⅓ Fortune 500 https://kafka.apache.org/powered-by Linkedin Netflix Yahoo Twitter Goldman Sachs
- 31. Other features Stream Aggregation Generate new topics Complex data pipelines Dynamic data processor KSQL Real time streaming with sql language
- 32. Other features - Stream
- 33. Other features - Stream
- 34. Other features - Stream
- 35. Use cases - Data replication App1 App2
- 36. Use cases - Messaging App1 App2
- 37. Use cases - Messaging App1 App2 App3
- 38. Use cases - Messaging App1 App2 App3
- 39. User activity Analysis / Dashboard
- 40. ETL BI
- 41. Events, Stream, Client Join MS1 MS2
- 42. Events, Stream, Client Join
- 43. Questions