Paris Kafka Meetup - How to develop with Kafka
1 like558 views
The document discusses the configuration and usage of Apache Kafka, focusing on producer and consumer properties such as async vs sync sending, batching, and partitions. It includes code snippets demonstrating Kafka producer and consumer implementations, error handling, retries, and manual partition assignments. The document also mentions the management of consumer groups and the importance of session timeouts and heartbeats.
![Configuration
config.put("partitioner.class", DefaultPartitioner.class.getName()
Implémenter un Partitioner
public interface Partitioner {
int partition(String topic,
Object key, byte[] keyBytes,
Object value, byte[] valueBytes, Cluster cluster);
}
Spécifier directement la partition cible
new ProducerRecord("my_topic", 0, "my_key", "my_value");](https://image.slidesharecdn.com/paris-kafka-meetup-may17-3-how-to-develop-160518203449/85/Paris-Kafka-Meetup-How-to-develop-with-Kafka-8-320.jpg)
![ConsumerRecords<Object, Object> records = consumer.poll(1000);
if( ! records.isEmpty() ) {
consumer.pause(consumer.assignment().toArray(new TopicPartition[0]));
Future<Boolean> future = executorService.submit(() -> {
records.forEach(record -> LOG.info("key={}, value={}", record.key(), record.value()));
return true;
});
Boolean isCompleted = false;
while(!isCompleted) {
try {
isCompleted = future.get(5, TimeUnit.SECONDS); // Wait before polling
} catch (TimeoutException e) {
consumer.poll(0); // heart-beat
} catch (CancellationException |ExecutionException | InterruptedException e) {
break;
}
}
consumer.resume(consumer.assignment().toArray(new TopicPartition[0]));
consumer.commitSync();
}
ExecutorService](https://image.slidesharecdn.com/paris-kafka-meetup-may17-3-how-to-develop-160518203449/85/Paris-Kafka-Meetup-How-to-develop-with-Kafka-18-320.jpg)
Paris Kafka Meetup - How to develop with Kafka
- 1. Paris Apache Kafka Meetup Florian HUSSONNOIS Zenika @fhussonnois
- 2. Async, Sync, Batch, Partitioner et Retries
- 3. Properties config = new Properties(); config.put("bootstrap.servers", "localhost:9092"); KafkaProducer<String, String> producer = new KafkaProducer(config); ProducerRecord record = new ProducerRecord("my_topic", "my_key", "my_value"); producer.send(record); producer.close();
- 4. L’appel à la méthode send()est asynchrone et retourne immédiatement Le message est ajouté à un buffer avant d’être envoyé //... config.put("batch.size", 16384); config.put("linger.ms", 1); //... Latence entre chaque transmission de messages Taille maximum d’un batch
- 5. List<ProducerRecord> batchRecords = new ArrayList<>(); //... for(ProducerRecord record : batchRecords) producer.send(record); producer.flush(); producer.close(); Force l’envoi des messages et bloque jusqu’à leur complétion
- 6. Future<RecordMetadata> future = producer.send(record); RecordMetadata metadata = future.get(); // BLOCK LOG.info("message sent to topic {}, partition {}, offset {}", metadata.topic(), metadata.partition(), metadata.offset());
- 7. ProducerRecord record = new ProducerRecord("my_topic", "my_key", "my_value"); Future<RecordMetadata> future = producer.send(record, (metadata, e) -> { if(e != null) LOG.info("Message sent to topic {}, partition {}, offset {}", metadata.topic(), metadata.partition(), metadata.offset()); else LOG.error("Damn it!", e); });
- 8. Configuration config.put("partitioner.class", DefaultPartitioner.class.getName() Implémenter un Partitioner public interface Partitioner { int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster); } Spécifier directement la partition cible new ProducerRecord("my_topic", 0, "my_key", "my_value");
- 9. Acknowledgments config.put("ack", "all "); // plus lent, messages répliqués par tous les ISR Le Producer peut rejouer automatiquement les messages en erreurs config.put("retries", "0 "); // désactivé /! Peut provoquer des doublons (At-Least Once) /! Peut changer l’ordre de publication des messages
- 10. Event Loop, Polling Model, Offset et Group Management
- 11. Properties config = new Properties(); config.put("bootstrap.servers", "localhost:9092"); KafkaConsumer<Object, Object> consumer = new KafkaConsumer<>(config); consumer.subscribe(Arrays.asList("topic1, topic2")); while(true) { ConsumerRecords<Object, Object> records = consumer.poll(1000); records.forEach(record -> LOG.info("key={}, value={}", record.key(), record.value())); } Event Loop, Polling Model
- 12. Properties config = new Properties(); config.put("bootstrap.servers", "localhost:9092"); config.put("enable.auto.commit", false); // désactive auto-commit config.put("auto.commit.interval.ms", 100); KafkaConsumer<Object, Object> consumer = new KafkaConsumer<>(config); consumer.subscribe(Arrays.asList("topic1, topic2")); while(true) { ConsumerRecords<Object, Object> records = consumer.poll(1000); records.forEach(record -> LOG.info("key={}, value={}", record.key(), record.value())); consumer.commitAsync(); } }
- 13. while(true) { ConsumerRecords<Object, Object> records = consumer.poll(1000); consumer.commitSync(); // Commit offsets before processing messages. records.forEach(record -> LOG.info("key={}, value={}", record.key(), record.value())); }
- 15. Properties config = new Properties(); config.put("bootstrap.servers", "localhost:9092"); config.put("group.id", "my_group"); KafkaConsumer<Object, Object> consumer = new KafkaConsumer<>(config); consumer.subscribe(Arrays.asList("topic1, topic2"));
- 16. KafkaConsumer<Object, Object> consumer = new KafkaConsumer<>(config); consumer.subscribe(Arrays.asList("topic1, topic2"), new ConsumerRebalanceListener() { @Override public void onPartitionsRevoked(Collection<TopicPartition> partitions) { //do some stuff } @Override public void onPartitionsAssigned(Collection<TopicPartition> partitions) { //do some stuff } });
- 17. Chaque consumer d’un groupe doit notifier le coordinateur Uniquement possible sur un appel aux méthodes poll, commit, etc. Déclenché si un consumer rejoint ou quitte un groupe L’opération de « rebalance » est impactée par les paramètres : • session.timeout.ms (30 secondes) • heartbeat.interval.ms Rebalance intempestif en cas de traitement d’un message trop long
- 18. ConsumerRecords<Object, Object> records = consumer.poll(1000); if( ! records.isEmpty() ) { consumer.pause(consumer.assignment().toArray(new TopicPartition[0])); Future<Boolean> future = executorService.submit(() -> { records.forEach(record -> LOG.info("key={}, value={}", record.key(), record.value())); return true; }); Boolean isCompleted = false; while(!isCompleted) { try { isCompleted = future.get(5, TimeUnit.SECONDS); // Wait before polling } catch (TimeoutException e) { consumer.poll(0); // heart-beat } catch (CancellationException |ExecutionException | InterruptedException e) { break; } } consumer.resume(consumer.assignment().toArray(new TopicPartition[0])); consumer.commitSync(); } ExecutorService
- 19. Se positionner à un offset spécifique consumer.seek(new TopicPartition("my_topic", 0), 42); consumer.seekToEnd(new TopicPartition("my_topic", 0)); consumer.seekToBeginning(new TopicPartition("my_topic", 0)); Assignements manuel consumer.assign(Arrays.asList(new TopicPartition("my_topic", 0))); Obtenir les métriques consumer.metrics();
- 20. Nous recrutons ! jobs@zenika.com @ZenikaIT Prochain Meetup le