What to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy
0 likes1,859 views
The document outlines solutions for handling Out of Memory (OOM) errors in Kafka Streams applications by implementing resource limits and heap configurations in Kubernetes. It discusses various JVM and RocksDB memory management techniques, including the use of specific Java parameters and dynamic memory allocation to prevent unbounded memory usage. Additionally, it provides example code snippets and links to further resources on memory tuning and analysis.
![Example app
8
Simple Kafka Streams topology
Application.java
public static void main(String[] args) {
var builder = new StreamsBuilder();
var stream = builder.stream(INPUT_TOPIC);
builder.addStateStore(Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(STATE_STORE_NAME));
var persistedStream = stream.transformValues((readOnlyKey, val) -> {
...
stateStore.put(readOnlyKey, val);
...
}, STATE_STORE_NAME);
persistedStream.foreach((key, val) -> ...);
var topology = builder.build();
var kafkaStreams = new KafkaStreams(topology);
runApp(kafkaStreams);
}](https://image.slidesharecdn.com/lt0320230516-ksl23-raiffeisenbank-serebryanskiyandrey-230602141916-c29db769/85/What-to-do-if-Your-Kafka-Streams-App-Gets-OOMKilled-with-Andrey-Serebryanskiy-8-320.jpg)
![Example app
9
Simple Kafka Streams topology
Application.java
public static void main(String[] args) {
var builder = new StreamsBuilder();
var stream = builder.stream(INPUT_TOPIC);
builder.addStateStore(Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(STATE_STORE_NAME));
var persistedStream = stream.transformValues((readOnlyKey, val) -> {
...
stateStore.put(readOnlyKey, val);
...
}, STATE_STORE_NAME);
persistedStream.foreach((key, val) -> ...);
var topology = builder.build();
var kafkaStreams = new KafkaStreams(topology);
runApp(kafkaStreams);
}](https://image.slidesharecdn.com/lt0320230516-ksl23-raiffeisenbank-serebryanskiyandrey-230602141916-c29db769/85/What-to-do-if-Your-Kafka-Streams-App-Gets-OOMKilled-with-Andrey-Serebryanskiy-9-320.jpg)
![Example app
10
Simple Kafka Streams topology
Application.java
public static void main(String[] args) {
var builder = new StreamsBuilder();
var stream = builder.stream(INPUT_TOPIC);
builder.addStateStore(Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(STATE_STORE_NAME));
var persistedStream = stream.transformValues((readOnlyKey, val) -> {
...
stateStore.put(readOnlyKey, val);
...
}, STATE_STORE_NAME);
persistedStream.foreach((key, val) -> logMessageCount());
var topology = builder.build();
var kafkaStreams = new KafkaStreams(topology);
runApp(kafkaStreams);
}](https://image.slidesharecdn.com/lt0320230516-ksl23-raiffeisenbank-serebryanskiyandrey-230602141916-c29db769/85/What-to-do-if-Your-Kafka-Streams-App-Gets-OOMKilled-with-Andrey-Serebryanskiy-10-320.jpg)
![Example app
11
Simple Kafka Streams topology
Application.java
public static void main(String[] args) {
var builder = new StreamsBuilder();
var stream = builder.stream(INPUT_TOPIC);
builder.addStateStore(Stores.keyValueStoreBuilder(
Stores.persistentKeyValueStore(STATE_STORE_NAME));
var persistedStream = stream.transformValues((readOnlyKey, val) -> {
...
stateStore.put(readOnlyKey, val);
...
}, STATE_STORE_NAME);
persistedStream.foreach((key, val) -> logMessageCount());
var topology = builder.build();
var kafkaStreams = new KafkaStreams(topology);
runApp(kafkaStreams);
}](https://image.slidesharecdn.com/lt0320230516-ksl23-raiffeisenbank-serebryanskiyandrey-230602141916-c29db769/85/What-to-do-if-Your-Kafka-Streams-App-Gets-OOMKilled-with-Andrey-Serebryanskiy-11-320.jpg)
![Example app
12
Simple Kafka Streams topology
Application.java
public static void main(String[] args) {
...
runApp(kafkaStreams);
}
Please find full application code here:
https://github.com/a-serebryanskiy/kafka-streams-oom-killed](https://image.slidesharecdn.com/lt0320230516-ksl23-raiffeisenbank-serebryanskiyandrey-230602141916-c29db769/85/What-to-do-if-Your-Kafka-Streams-App-Gets-OOMKilled-with-Andrey-Serebryanskiy-12-320.jpg)
What to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy
- 1. What to Do if Your Kafka Streams App Gets OOMKilled? Andrey Serebryanskiy
- 2. { Andrey Serebryanskiy Streaming Platform Owner at Raiffeisen Bank What to Do if Your Kafka Streams App Gets OOMKilled?
- 3. The problem 3 Kafka Kafka Streams App kubernetes RocksDB
- 4. What is the problem with this app?
- 5. Launch with resource limits 5 helm/templates/deployment.yaml ... resources: limits: memory: 256Mi requests: memory: 128Mi command: - java args: - -jar - app.jar ...
- 6. How to check if it is OOMKilled? 6 kubectl describe pod your-pod-name -n your-namespace Name: your-pod-name ... Containers: app: ... Last State: Terminated Reason: OOMKilled Exit Code: 137
- 7. Example app
- 8. Example app 8 Simple Kafka Streams topology Application.java public static void main(String[] args) { var builder = new StreamsBuilder(); var stream = builder.stream(INPUT_TOPIC); builder.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STATE_STORE_NAME)); var persistedStream = stream.transformValues((readOnlyKey, val) -> { ... stateStore.put(readOnlyKey, val); ... }, STATE_STORE_NAME); persistedStream.foreach((key, val) -> ...); var topology = builder.build(); var kafkaStreams = new KafkaStreams(topology); runApp(kafkaStreams); }
- 9. Example app 9 Simple Kafka Streams topology Application.java public static void main(String[] args) { var builder = new StreamsBuilder(); var stream = builder.stream(INPUT_TOPIC); builder.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STATE_STORE_NAME)); var persistedStream = stream.transformValues((readOnlyKey, val) -> { ... stateStore.put(readOnlyKey, val); ... }, STATE_STORE_NAME); persistedStream.foreach((key, val) -> ...); var topology = builder.build(); var kafkaStreams = new KafkaStreams(topology); runApp(kafkaStreams); }
- 10. Example app 10 Simple Kafka Streams topology Application.java public static void main(String[] args) { var builder = new StreamsBuilder(); var stream = builder.stream(INPUT_TOPIC); builder.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STATE_STORE_NAME)); var persistedStream = stream.transformValues((readOnlyKey, val) -> { ... stateStore.put(readOnlyKey, val); ... }, STATE_STORE_NAME); persistedStream.foreach((key, val) -> logMessageCount()); var topology = builder.build(); var kafkaStreams = new KafkaStreams(topology); runApp(kafkaStreams); }
- 11. Example app 11 Simple Kafka Streams topology Application.java public static void main(String[] args) { var builder = new StreamsBuilder(); var stream = builder.stream(INPUT_TOPIC); builder.addStateStore(Stores.keyValueStoreBuilder( Stores.persistentKeyValueStore(STATE_STORE_NAME)); var persistedStream = stream.transformValues((readOnlyKey, val) -> { ... stateStore.put(readOnlyKey, val); ... }, STATE_STORE_NAME); persistedStream.foreach((key, val) -> logMessageCount()); var topology = builder.build(); var kafkaStreams = new KafkaStreams(topology); runApp(kafkaStreams); }
- 12. Example app 12 Simple Kafka Streams topology Application.java public static void main(String[] args) { ... runApp(kafkaStreams); } Please find full application code here: https://github.com/a-serebryanskiy/kafka-streams-oom-killed
- 13. So you have your app OOMKilled
- 14. Launch with resource limits 14 helm/templates/deployment.yaml ... resources: limits: memory: 256Mi requests: memory: 128Mi command: - java args: - -jar - app.jar ...
- 15. Let’s add heap limits 15 helm/templates/deployment.yaml ... resources: limits: memory: 256Mi requests: memory: 128Mi command: - java args: - -XshowSettings:VM - -XX:MinRAMPercentage=50.0 - -jar - app.jar ...
- 16. Let’s add heap limits 16 helm/templates/deployment.yaml ... resources: limits: memory: 256Mi requests: memory: 128Mi command: - java args: - -XshowSettings:VM - -XX:MinRAMPercentage=50.0 - -jar - app.jar ... https://www.baeldung.com/java-jvm-parameters-rampercentage VM settings: Max. Heap Size (Estimated): 121.81M Property settings: java.version = 11.0.12 Turns out 50.0 is already a default value
- 17. App memory performance 17 Taken from grafana container memory limit container memory usage jvm memory 100 mb
- 19. Kafka Streams app memory 19 JVM Heap + RocksDB Confluent article about Kafka Streams memory tunning: https://docs.confluent.io/platform/current/streams/developer-guide/memory-mgmt.html State Store Kafka Streams App put(key, val)
- 20. Kafka Streams app memory 20 JVM Heap + RocksDB Kafka Streams App CachingKeyValueStore - ThreadCache context.cache() Indexes bloom filters block cache OS page cache memtable State Store - RocksDB ! These items size depends on number of unique keys JVM Heap Not controlled by JVM RocksDB memory details: https://github.com/facebook/rocksdb/wiki/Memory-usage-in-RocksDB put(key, val) native put(key, val)
- 21. How to fix unbounded RocksDB memory usage? 21 Application.java properties.put(StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG, ”your.package.BoundedMemoryRocksDBConfig"); your.package.BoundedMemoryRocksDBConfig.java @Override public void setConfig(..., Options options, ...) { BlockBasedTableConfig tableConfig = options.tableFormatConfig(); Cache cache = new LRUCache(computeTotalRocksDbMem(), -1, false); tableConfig.setBlockCache(cache); tableConfig.setCacheIndexAndFilterBlocks(true); options.setWriteBufferManager(writeBufferManager); ... }
- 22. How to fix unbounded RocksDB memory usage? 22 Application.java properties.put(StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG, ”your.package.BoundedMemoryRocksDBConfig"); your.package.BoundedMemoryRocksDBConfig.java @Override public void setConfig(..., Options options, ...) { BlockBasedTableConfig tableConfig = options.tableFormatConfig(); Cache cache = new LRUCache(computeTotalRocksDbMem(), -1, false); tableConfig.setBlockCache(cache); tableConfig.setCacheIndexAndFilterBlocks(true); options.setWriteBufferManager(writeBufferManager); ... }
- 23. How to fix unbounded RocksDB memory usage? 23 Application.java properties.put(StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG, ”your.package.BoundedMemoryRocksDBConfig"); your.package.BoundedMemoryRocksDBConfig.java @Override public void setConfig(..., Options options, ...) { BlockBasedTableConfig tableConfig = options.tableFormatConfig(); Cache cache = new LRUCache(computeTotalRocksDbMem(), -1, false); tableConfig.setBlockCache(cache); tableConfig.setCacheIndexAndFilterBlocks(true); } Pay attention to the number of stores and partitions: https://docs.confluent.io/platform/current/streams/developer-guide/memory-mgmt.html#rocksdb
- 24. How to compute my RocksDB memory?
- 25. Dynamic memory allocation 25 your.package.BoundedMemoryRocksDBConfig.java private static long computeTotalRocksDbMem() { long totalContainerMemoryBytes = getEnv("CONTAINER_MEMORY_LIMIT")); double osPercentage = getEnv("OS_MEMORY_PERCENTAGE")); long offHeapSizeMb = getEnv("OFF_HEAP_SIZE_MB")); long maxHeapSizeMb = getEnv("MAX_HEAP_SIZE_MB")); long jvmMemoryBytes = (offHeapSizeMb + maxHeapSizeMb) * 1024 * 1024; return totalContainerMemoryBytes * (1 - osPercentage)) - jvmMemoryBytes; }
- 26. Dynamic memory allocation 26 your.package.BoundedMemoryRocksDBConfig.java private static long computeTotalRocksDbMem() { long totalContainerMemoryBytes = getEnv("CONTAINER_MEMORY_LIMIT")); double osPercentage = getEnv("OS_MEMORY_PERCENTAGE")); long offHeapSizeMb = getEnv("OFF_HEAP_SIZE_MB")); long maxHeapSizeMb = getEnv("MAX_HEAP_SIZE_MB")); long jvmMemoryBytes = (offHeapSizeMb + maxHeapSizeMb) * 1024 * 1024; return totalContainerMemoryBytes * (1 - osPercentage)) - jvmMemoryBytes; }
- 27. Dynamic memory allocation 27 About using container props as env vars: https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/ helm/templates/deployment.yaml ... env: - name: CONTAINER_MEMORY_LIMIT valueFrom: resourceFieldRef: containerName: app resource: limits.memory ...
- 28. Dynamic memory allocation 28 your.package.BoundedMemoryRocksDBConfig.java private static long computeTotalRocksDbMem() { long totalContainerMemoryBytes = getEnv("CONTAINER_MEMORY_LIMIT")); double osPercentage = getEnv("OS_MEMORY_PERCENTAGE")); long offHeapSizeMb = getEnv("OFF_HEAP_SIZE_MB")); long maxHeapSizeMb = getEnv("MAX_HEAP_SIZE_MB")); long jvmMemoryBytes = (offHeapSizeMb + maxHeapSizeMb) * 1024 * 1024; return totalContainerMemoryBytes * (1 - osPercentage)) - jvmMemoryBytes; }
- 29. Dynamic memory allocation 29 helm/templates/deployment.yaml env: - name: OS_MEMORY_PERCENTAGE value: "0.1" - name: OFF_HEAP_SIZE_MB value: "128" - name: MAX_HEAP_SIZE_MB value: "{{ .Values.heapSizeMb }}"
- 30. Dynamic memory allocation 30 your.package.BoundedMemoryRocksDBConfig.java private static long computeTotalRocksDbMem() { long totalContainerMemoryBytes = getEnv("CONTAINER_MEMORY_LIMIT")); double osPercentage = getEnv("OS_MEMORY_PERCENTAGE")); long offHeapSizeMb = getEnv("OFF_HEAP_SIZE_MB")); long maxHeapSizeMb = getEnv("MAX_HEAP_SIZE_MB")); long jvmMemoryBytes = (offHeapSizeMb + maxHeapSizeMb) * 1024 * 1024; return totalContainerMemoryBytes * (1 - osPercentage)) - jvmMemoryBytes; }
- 31. Dynamic memory allocation 31 helm/templates/deployment.yaml env: - name: OS_MEMORY_PERCENTAGE value: "0.1" # computed it based on the jcmd output - name: OFF_HEAP_SIZE_MB value: "128" - name: MAX_HEAP_SIZE_MB value: "{{ .Values.heapSizeMb }}"
- 32. If you would like to analyze non-heap JVM mem 32 1. Make sure you use JDK (not JRE) Docker image 2. Add to JVM args -XX:NativeMemoryTracking=summary 3. Execute command in container shell: helm/templates/deployment.yaml command: - java args: - -XX:NativeMemoryTracking=summary - -jar - app.jar bash kubectl exec pod/your-pod-name -n your-namespace –it -- /bin/bash -c “jcmd 1 VM.native_memory”
- 33. Dynamic memory allocation 33 your.package.BoundedMemoryRocksDBConfig.java private static long computeTotalRocksDbMem() { long totalContainerMemoryBytes = getEnv("CONTAINER_MEMORY_LIMIT")); double osPercentage = getEnv("OS_MEMORY_PERCENTAGE")); long offHeapSizeMb = getEnv("OFF_HEAP_SIZE_MB")); long maxHeapSizeMb = getEnv("MAX_HEAP_SIZE_MB")); long jvmMemoryBytes = (offHeapSizeMb + maxHeapSizeMb) * 1024 * 1024; return totalContainerMemoryBytes * (1 - osPercentage)) - jvmMemoryBytes; }
- 34. Dynamic memory allocation 34 helm/templates/deployment.yaml env: - name: OFF_HEAP_SIZE_MB value: "128" - name: MAX_HEAP_SIZE_MB value: "{{ .Values.heapSizeMb }}” args: - -Xmx{{ .Values.heapSizeMb }}m - -jar - app.jar helm/values.yaml heapSizeMb: 64
- 35. If you would like to profile your app 35 1) helm/templates/deployment.yaml command: - java args: - -Dcom.sun.management.jmxremote - -Dcom.sun.management.jmxremote.port=13089 - -Dcom.sun.management.jmxremote.ssl=false - -Dcom.sun.management.jmxremote.local.only=false - -Dcom.sun.management.jmxremote.authenticate=false - -Dcom.sun.management.jmxremote.rmi.port=13089 - -Djava.rmi.server.hostname=localhost - -jar - app.jar ports: - containerPort: 13089 name: jmx protocol: TCP 2) kubectl port-forward pod/your-pod-name -n your-namespace 13089:13089
- 36. App memory performance (after fix) 36 Taken from grafana
- 37. Limitations are not the only way! 37
- 38. Links and materials 38 • How JVM analyze memory inside Docker container– https://merikan.com/2019/04/jvm-in-a-container/#java-10 • How to use jcmd to analyze non-heap memory– https://www.baeldung.com/native-memory-tracking-in-jvm • Why use container_memory_working_set_bytes instead of container_memory_usage_bytes https://blog.freshtracks.io/a-deep-dive-into-kubernetes-metrics-part-3-container-resource-metrics-361c5ee46e6 • How RocksDB store works with Kafka Streams: https://www.confluent.io/blog/how-to-tune-rocksdb-kafka-streams-state-stores-performance/ • Linux memory controller for cgroups https://lwn.net/Articles/432224/