Running Kafka as a Native Binary Using GraalVM with Ozan Günalp
0 likes1,397 views
The document discusses running Kafka as a native binary using GraalVM, highlighting benefits such as reduced startup times and lower memory overhead. It outlines the use of different container images for Kafka testing and development, as well as the features of Quarkus that enhance developer productivity. Additionally, it touches on future enhancements planned for Kafka-native, including support for multiple Kafka versions and experimental JMX support.
Running Kafka as a Native Binary Using GraalVM with Ozan Günalp
- 1. Running Kafka as a Native Binary using GraalVM Ozan Gunalp, Red Hat
- 2. Ozan Gunalp, Red Hat – Quarkus @ozangunalp Running Kafka as a Native Binary using GraalVM
- 3. The Problem Kafka is fast! 3 Can we make it faster? Reducing broker startup time as much as possible sub second startup time
- 4. Integration Tests with Kafka KafkaContainer - Confluent platform image, provided by org.testcontainers:kafka - Zookeeper & Kraft-mode - Not compatible with random bootstrap.servers port StrimziKafkaContainer - Strimzi image, provided by io.strimzi:strimzi-test-container - Zookeeper & Kraft-mode - Can be used with random port - Cluster support - Toxiproxy support to simulate network conditions 4 Both are slow to startup Kraft with format-storage uses another Java process to create log-dir
- 6. Quarkus 6 Frictionless development Optimized for Distributed Architectures
- 7. Quarkus Developer Joy 7 Dev Mode with live reload, Dev services and Continuous testing Unified Configuration Customizable defaults
- 8. Quarkus Kube-Native 8 Low memory overhead + Fast startup time + Small container size = Increase the deployment density Observability & Resilience ❤
- 10. kafka-native 10 Kafka broker embedded in a Quarkus application Default configuration for dev/test purposes Completely configurable Compiled to native binary using GraalVM
- 11. GraalVM Native Image Run the same workload on smaller machines on the cloud 11 🚀 🔐 🗜 Smaller size images – no JVM included Minimal attack surface 🍃 Smaller memory footprint Faster startup time – No warmup
- 12. native-image compilation Reachability analysis Build-time Class Initialization Heap snapshotting Metadata for Runtime Reflection, Security Providers 12
- 13. Compiling Kafka to native Netty GraalVM support inherited from Quarkus Security Providers Plain, Scram, OAuth, GSSAPI Substitutions No Random init on static fields, No thread initialization Bytecode manipulation No JMX Support – YammerMetrics, MBean 13
- 14. kafka-native Quarkus Dev Service 14 Startup time RSS at startup quay.io/strimzi-test-container/test- container @0.103.0-kafka-3.3.2 ~3s 370Mb vectorized/redpanda @v22.3.4 ~200ms 135Mb quay.io/ogunalp/kafka-native @0.2.0-kafka-3.4.0 ~120ms 45Mb quarkus.kafka.devservice.provider=kafka-native
- 15. kafka-native-test-container <!-- https://mvnrepository.com/artifact/com.ozangunalp/kafka-native-test-container --> <dependency> <groupId>com.ozangunalp</groupId> <artifactId>kafka-native-test-container</artifactId> <version>0.2.0</version> </dependency> 15 try (var container = new KafkaNativeContainer()) { container.start(); String bootstrapServers = container.getBootstrapServers(); // create topics // produce // consume }
- 16. kafka-native-test-container 16 String clusterId = Uuid.randomUuid().toString(); String quorumVotes = "1@broker1:9094"; var controllerAndBroker = new KafkaNativeContainer(); var brokerOnly = new KafkaNativeContainer(); var common = Map.of( "SERVER_CLUSTER_ID", clusterId, "KAFKA_CONTROLLER_QUORUM_VOTERS", quorumVotes); common.forEach(controllerAndBroker::addEnv); common.forEach(brokerOnly::addEnv); controllerAndBroker.withNetworkAliases(broker1); controllerAndBroker.withNetwork(network); controllerAndBroker.addEnv("SERVER_HOST", "broker1"); controllerAndBroker.addEnv("KAFKA_BROKER_ID", "1"); brokerOnly.withNetworkAliases(broker2); brokerOnly.withNetwork(network); brokerOnly.addEnv("SERVER_HOST", "broker2"); brokerOnly.addEnv("KAFKA_BROKER_ID", "2"); brokerOnly.addEnv("KAFKA_PROCESS_ROLES", "broker"); Startables.deepStart(controllerAndBroker, brokerOnly); var bs = brokerOnly.getBootstrapServers(); // produce, consume
- 17. kafka-native on the Cloud 17 Azure Container Instances Single-node 1vCPU - 1Gb ~$1.30 per day (us-central) 1rps, 100rps, 1000rps, 5000rps This is not a benchmark
- 18. Caveats Good for tests and dev Not production-ready Classes are linked statically: limited strategy classes GC performance – only Serial GC in GraalVM CE for now Lacks managed beans / JMX for observability Experimental support coming in GraalVM 23 18
- 19. What’s Next? 19 Support Zookeeper (Thanks to @kwall) Support multiple Kafka versions (Thanks to @franvila) Multi-arch images (ARM support) Move to Smallrye organization Experimental JMX Support https://github.com/ozangunalp/kafka-native
- 21. vs Redpanda kafka-native IS Kafka upstream code Redpanda is more than a Kafka-compatible broker Schema registry, HTTP proxy and UI console Redpanda behaves slightly differently Redpanda transactions do not support complete Kafka transactions protocol (?) 21
- 22. Strategy config principal.builder.class, default: DefaultKafkaPrincipalBuilder replica.selector.class sasl.client.callback.handler.class sasl.login.callback.handler.class sasl.login.class sasl.server.callback.handler.class security.providers ssl.engine.factory.class alter.config.policy.class.name authorizer.class.name client.quota.callback.class create.cluster.link.policy.class.name create.topic.policy.class.name kafka.metrics.reporters metric.reporters 22