Devoxx Morocco 2016 - Microservices with Kafka
3 likes872 views
This document discusses using microservices with Kafka. It describes how Kafka can be used to connect microservices for asynchronous communication. It outlines various features of Kafka like high throughput, replication, partitioning, and how it can provide reliability. Examples are given of how microservices could use Kafka for logging, filtering messages, and dispatching to different topics. Performance benefits of Kafka are highlighted like scalability and ability to handle high volumes of messages.
Devoxx Morocco 2016 - Microservices with Kafka
- 1. Microservices with Kafka László-Róbert ALBERT Dan BALESCU Casablanca • 02 November 2016
- 2. Róbert Dan
- 3. ING Software Development Center Bucharest 3
- 4. ING Software Development Center Bucharest 4 IT 5,7% of GDP with 100k ppl Agriculture 4,4% of GDP with 1.8 million ppl
- 6. 6
- 7. 7
- 14. 14
- 15. 500 million messages / day 15
- 16. 500 million messages / day 700 billion messages / day 16
- 17. 500 million messages / day 700 billion messages / day 1.4 trillion messages / day 17
- 18. 500 million messages / day 700 billion messages / day 1.4 trillion messages / day 18
- 19. Messaging Brokers Features 19 Kafka ActiveMQ RabbitMQ
- 20. Messaging Brokers Features 20 Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc Kafka ActiveMQ RabbitMQ
- 21. Messaging Brokers Features 21 Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc Kafka ActiveMQ RabbitMQ Java Messaging System
- 22. Messaging Brokers Features 22 Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc Kafka ActiveMQ RabbitMQ Java Messaging System Java Authentication and Authorization Service Simple Authentication and Security Layer
- 23. Messaging Brokers Features 23 Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc Kafka ActiveMQ RabbitMQ Java Messaging System Java Authentication and Authorization Service Simple Authentication and Security Layer Transactions
- 24. Messaging Brokers Features 24 Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc Kafka ActiveMQ RabbitMQ Java Messaging System Java Authentication and Authorization Service Simple Authentication and Security Layer Transactions Filtering
- 25. Messaging Brokers Features 25 Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc Kafka ActiveMQ RabbitMQ Java Messaging System Java Authentication and Authorization Service Simple Authentication and Security Layer Transactions Filtering Persistence
- 27. 27 Messaging Brokers Performance Message producing Message consumption LinkedIn – Kafka: a Distributed Messaging System for Log Processing
- 29. 29 Usecases of Kafka Consumer Microservice A Microservice B Transport API Producer
- 30. 30 Usecases of Kafka Consumer Microservice A Microservice B Transport API Producer Consumer Producer
- 31. 31 Usecases of Kafka Microservice A Microservice B Transport API Producer Consumer Microservice C Consumer
- 32. 32 System Availability & Performance
- 33. 33 System Availability & Performance
- 34. 34 Scalability
- 35. 35 Scalability
- 36. 36 Usecases of Kafka Microservice A ELK stackKafka ConsumerKafka Appender K A F K A
- 37. © ING 2016 37 Producer // build a 'default' factory based on your-implicit-kafka-producer.properties // - the properties file contains the list of brokers to connect to ProducerFactory factory = ProducerFactory.builder().build(); // create a message producer with a Payload that sends the messages to topic ‘test’ Producer<Message<Payload>> producer = factory.createMessageProducer("test", Payload.class); // send a Message { header: Map<String, String>; payload: Payload } producer.send( Message.<Payload> builder() .header("applicationId", "sample-producer") .header("timestamp", now()) .payload(new Payload("/some/file/path")) .build() );
- 38. 38 Producer Native Kafka Producer API your-implicit-producer.properties your-security.properties
- 39. 39 Consumer // build a 'default' factory – your-implicit-kafka-consumer.properties // - the properties contains the list of brokers from which it will receive messages ConsumerFactory factory = ConsumerFactory.builder().build(); // obtain a consumer instance that can process Message<Payload> messages Consumer<Message<Payload>> consumer = factory.createMessageConsumer(Payload.class); // prepare the message (Message { header: Map<String, String>; payload: Payload } ) handler ReceiverHandler<Message<Payload>> messageReceiver = new ReceiverHandler<Message<Payload>>() { @Override public void onReceive(Message<Payload> message) { System.out.printf("message received: %s%n", message); } }; // register the handler to receive messages from the topic named “test” consumer.registerHandler("test", messageReceiver); © ING 2016
- 40. 40 Consumer Native Kafka Consumer API your-implicit-consumer.properties your-security.properties
- 41. 41 KafkaAppender ELK KAFKA Microservice A Logback File Appender Kafka Appender Kafka Producer Logstash Fordwarder
- 42. 43 Mutual TLS - your-security.properties # The location of the key store file. kafka.tls.keystore.location=/full/path/to/a/keystore.jks # The store password for the key store file. kafka.tls.keystore.password=some-good-password # The file format of the key store file. kafka.tls.keystore.type=JKS # The password of the private key in the key store file kafka.tls.key.password=the-private-keys-password # The location of the trust store file. kafka.tls.truststore.location=/path/to/truststore.p12 # The password for the trust store file. kafka.tls.truststore.password=some-other-good-password # The file format of the trust store file kafka.tls.truststore.type=PKCS12 © ING 2016
- 43. 44 Serialize Producer 1 Metadata N Metadata 1 N Transformer Serializer MetadataMetadata KAFKA1 2 3
- 44. 45 Filter Microservice A Kafka Producer Microservice B Kafka Consumer KAFKA Topic A Kafka Consumer Kafka Producer Metadata Topic B Filter
- 45. Transformer 46 Transformer Microservice A Kafka Producer Microservice B Kafka Consumer KAFKA Topic A Topic B 1 Kafka Consumer Kafka Producer Metadata 1 2 1 2 2
- 46. 47 Forward & Dispatch Microservice A Kafka Producer Microservice B Kafka Consumer KAFKA Topic A Topic C Kafka Consumer Metadata Topic B Topic N Kafka Producer 1 1 1 1 Microservice D Kafka Consumer Microservice Z Kafka Consumer Forward & Dispatch
- 47. 48 Replay 0 1 2 3 i NN-1 beginning i end Microservice A
- 51. Use microservices with Kafka … … code ahead of the curve László-Róbert ALBERT Dan BALESCU Casablanca • 02 November 2016 @danbalescu https://www.linkedin.com/in/albertlaszlorobert