Guaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, Nutanix
0 likes766 views
The document discusses utilizing Kafka with Node.js for guaranteed event delivery, highlighting advantages such as real-time updates through webhooks and an easy subscription interface. It elaborates on design considerations for event delivery methods, including at-most-once, at-least-once, and exactly-once semantics, and provides implementation details for both producing and consuming messages. Key takeaways emphasize the decoupling of services using Kafka, the importance of retries for failed events, and the benefits of asynchronous processing with Node.js.
![PUT /subscriptions/subscribers/<id>
{
"events": ["PAYMENT_METHOD_EXPIRING"],
"subscriber": ”CLOUD_INFRA",
"method": "POST",
"url": "https://endpoint",
"auth": "JWT",
"audience": "subscriber-aud-url",
"email":"team@nutanix.com"
}
• Prerequisites
• Provides access to events
• Upsert API](https://image.slidesharecdn.com/amiteshmadhur-210928170301/85/Guaranteed-Event-Delivery-with-Kafka-and-NodeJS-Amitesh-Madhur-Nutanix-28-320.jpg)
![Produce to Ka;a
Endpoint
POST http://kafka-host:8082/topics/{topic-name}
Headers:
"Accept": "application/vnd.kafka.json.v2+json, application/vnd.kafka+json,
application/json”
"Content-Type": "application/vnd.kafka.json.v2+json"
Payload:
{
"records": [{
"key": "key-1",
"value": {...}
}]
}](https://image.slidesharecdn.com/amiteshmadhur-210928170301/85/Guaranteed-Event-Delivery-with-Kafka-and-NodeJS-Amitesh-Madhur-Nutanix-37-320.jpg)
![consumer.on('ready', function(arg) {
console.log('consumer ready.' + JSON.stringify(arg));
consumer.subscribe(['topic1', 'topic2']);
consumer.consume();
});
//starting the consumer
consumer.connect();](https://image.slidesharecdn.com/amiteshmadhur-210928170301/85/Guaranteed-Event-Delivery-with-Kafka-and-NodeJS-Amitesh-Madhur-Nutanix-41-320.jpg)
Guaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, Nutanix
- 1. Guaranteed event delivery with Ka1a and NodeJS A U G 2 0 2 1
- 6. Ka#a
- 7. Kafka
- 8. 1 User Management 5 Support 2 6 Licensing 3 Billing 7 API keys and Access Management 4 Onboarding 8 Many Automa=ons Tenant Management
- 9. • Polling, data are almost always old. • Most of the polling requests results in no change • IO overheard, even if data is cached • No Retry implementation needed • Webhooks are more opDmal • More real time, less chatty • SubscripDon, Delivery and Retry implementaDon Polling and Webhook
- 10. Node JS Data APIs that interact with database. Events To build event driven APIs/Microservices Non-blocking Async processing IO API Calls
- 12. Why Node JS Callback Complete • Web Servers • IntegraDon APIs • Frontend server • APIs with Database • Command line Apps • Webhooks • Real time IO, Web sockets
- 13. Worker Threads 200/OK { success: true, data: Array(1010000) }
- 14. 1 Enables Frontend services to directly access data. 2 3 API Authen=ca=on. 4 Click streaming. Both Node and Kafka follows similar scaling methodology. Advantages of Ka;a for Node JS
- 15. Subscribers
- 16. Subscribers
- 17. Subscribers
- 18. Subscribers
- 19. Subscribers
- 20. Design considera>ons • Onboarding instructions • Public key exchange • List of events
- 21. Design considera>ons • How do we deliver events? • at-most-once • at-least-once • exactly-once ü Onboarding instructions ü Public key exchange ü List of events
- 27. Design considera>ons ü How do we deliver events? ü at-most-once ü at-least-once ü exactly-once ü Onboarding instructions ü Public key exchange ü List of events • Easy subscripDon interface
- 28. PUT /subscriptions/subscribers/<id> { "events": ["PAYMENT_METHOD_EXPIRING"], "subscriber": ”CLOUD_INFRA", "method": "POST", "url": "https://endpoint", "auth": "JWT", "audience": "subscriber-aud-url", "email":"team@nutanix.com" } • Prerequisites • Provides access to events • Upsert API
- 30. Design considera>ons ü How do we deliver events? ü at-most-once ü at-least-once ü exactly-once ü Onboarding instructions ü Public key exchange ü List of events ü Easy subscripDon interface • Produce event
- 31. Produce to Ka;a const Kafka = require('node-rdkafka');
- 32. Produce to Ka;a const producer = new Kafka.Producer({ 'metadata.broker.list': 'broker-host:port’, 'dr_cb': true //delivery report callback });
- 33. Produce to Ka;a /* producer.produce(topic, partition, msg, key, timestamp, opaque) */ producer.on('ready', function() { const message = Buffer.from('Awesome message’); try { producer.produce ('topicName', null, message, 'Key-1', Date.now(), 'Key-1-opaque-token’); } catch (err) { console.error('A problem occurred:', err); } });
- 34. // Any errors? including connection errors producer.on('event.error', function(err) { console.error('Error from producer', err); }) producer.setPollInterval(100); Produce to Ka;a producer.on('delivery-report', function(err, report) { console.log('delivery-report:', report); });
- 35. Produce to Kafka // connect producer producer.connect();
- 36. Produce to Kafka const Kafka = require('node-rdkafka'); const producer = new Kafka.Producer({ 'metadata.broker.list': 'broker-host:port’, 'dr_cb': true //delivery report callback }); /* producer.produce(topic, partition, msg, key, timestamp, opaque) */ producer.on('ready', function() { const message = Buffer.from('Awesome message’); try { producer.produce ('topicName', null, message, 'Key-1', Date.now(), 'Key-1-opaque-token’); } catch (err) { console.error('A problem occurred:', err); } }); // Any errors? including connection errors producer.on('event.error', function(err) { console.error('Error from producer', err); }) producer.setPollInterval(100); producer.on('delivery-report', function(err, report) { console.log('delivery-report:', report); }); // connect producer producer.connect();
- 37. Produce to Ka;a Endpoint POST http://kafka-host:8082/topics/{topic-name} Headers: "Accept": "application/vnd.kafka.json.v2+json, application/vnd.kafka+json, application/json” "Content-Type": "application/vnd.kafka.json.v2+json" Payload: { "records": [{ "key": "key-1", "value": {...} }] }
- 39. Design considera>ons ü How do we deliver events? ü at-most-once ü at-least-once ü exactly-once ü Onboarding instructions ü Public key exchange ü List of events ü Easy subscripDon interface ü Produce event • Consume and process near real Dme
- 40. const consumer = new Kafka.KafkaConsumer({ 'metadata.broker.list': 'localhost:9092’, 'group.id': 'node-consumer’, 'enable.auto.commit': false });
- 41. consumer.on('ready', function(arg) { console.log('consumer ready.' + JSON.stringify(arg)); consumer.subscribe(['topic1', 'topic2']); consumer.consume(); }); //starting the consumer consumer.connect();
- 42. consumer.on('data', function(data) { startProcessing(data.value); }); // structure of data { value: Buffer.from('hi'), // message contents as a Buffer size: 2, // size of the message, in bytes topic: ‘topic', // topic the message comes from offset: 10, // offset the message was read from partition: 1, // partition the message was on key: ‘key-1', // key of the message if present timestamp: 1628388840187// timestamp of message creation }
- 44. Webhook const kafka = require('node-rdkafka'); const config = require('./consumerConfig') const consumer = new kafka.Consumer(config); consumer.on('data', (data) => { processData(data); }); consumer.connect(); const processData = async(data) => { // Takes 1 sec to process }; queued.max.messages.kbytes
- 47. Design considerations ü How do we deliver events? ü at-most-once ü at-least-once ü exactly-once ü Onboarding instrucDons ü Public key exchange ü List of events ü Easy subscription interface ü Produce event ü Consume and process near real Dme • Ordering and Retry
- 48. Per subscriber queue • Notify topic is partitioned by the number of total subscribers. • Each partition will keep relevant subscribed events. • After successful delivery, the consumer commits the offset. • Partitions are limited, scale issue.
- 49. Multiple consumer groups • Dispatcher does the ordering and produces the delivery topic. • Each notifier has subscribed to the topic with a unique consumer group id. • Manual commit after successful delivery.
- 50. • Each producers have their own topic. • Each events are keyed by username. • Manual commit after successful delivery.
- 51. • We get DNS error or TCP Error or TIMEOUT • Queue failed events for retry.
- 52. • Non 2xx response. • Queue failed events for retry. • Subscribers re-subscribe every time they have known error or downtime. • On subscription trigger retry.
- 53. • TIMEOUT • Queue failed events for retry.
- 54. • Save the state in database • Alert
- 55. Key Take-aways! ü How Ka#a enables Node JS. ü Introducing Kafka decouples services. ü Webhook is useful for delivering events, Webhooks can use Ka#a. ü Qualities of Node JS. IP, Data, Event and Async. ü Webhook with at-least-once delivery. ü Strategies of parallel processing and optimizing consumers. ü Sequencing and retry. ü Allow subscribers to resubscribe and trigger retry.
- 56. Thank you