The Dark and Dirty Side of Fixing Uneven Partitions with Olena Babenko & Olena Kutsenko
0 likes708 views
The document discusses strategies for partitioning data effectively to avoid issues such as uneven load distribution across partitions, which can lead to system performance problems like slower brokers and increased consumer lag. It outlines varying levels of complexity in approaches to manage uneven partitions, from simply increasing the number of partitions to using advanced techniques involving custom partitioners and potentially migrating to new topics for better balance and scalability. The importance of understanding data consumption patterns, resource allocation, and disaster recovery during these partitioning strategies is emphasized.
![olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Example
public static int partitionForKey(final byte[] serializedKey, final int numPartitions) {
if (serializedKey == "bananas🍌🍌") {
... do the dirty magic here ...
} else {
return Utils.toPositive(Utils.murmur2(serializedKey)) % (numPartitions - 1);
}
}](https://image.slidesharecdn.com/bs20-20230516-ksl23-babenkokutsenko-230602115643-3fd1aca0/85/The-Dark-and-Dirty-Side-of-Fixing-Uneven-Partitions-with-Olena-Babenko-Olena-Kutsenko-49-320.jpg)
![olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Example
public static int partitionForKey(final byte[] serializedKey, final int numPartitions) {
if (serializedKey == "bananas🍌🍌") {
... do the dirty magic here ...
} else {
return Utils.toPositive(Utils.murmur2(serializedKey)) % (numPartitions - 1);
}
}](https://image.slidesharecdn.com/bs20-20230516-ksl23-babenkokutsenko-230602115643-3fd1aca0/85/The-Dark-and-Dirty-Side-of-Fixing-Uneven-Partitions-with-Olena-Babenko-Olena-Kutsenko-50-320.jpg)
![olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Example
public static int partitionForKey(final byte[] serializedKey, final int numPartitions) {
if (serializedKey == "bananas🍌🍌") {
... do the dirty magic here ...
} else {
return Utils.toPositive(Utils.murmur2(serializedKey)) % (numPartitions - 1);
}
}](https://image.slidesharecdn.com/bs20-20230516-ksl23-babenkokutsenko-230602115643-3fd1aca0/85/The-Dark-and-Dirty-Side-of-Fixing-Uneven-Partitions-with-Olena-Babenko-Olena-Kutsenko-51-320.jpg)
![olena@aiven.io
@OlenaKutsenko aiven.io Olena Babenko:
Offset Translation
For r in records:
P = r.metadata.old_partition
If offsets[P] <= r.metadata.offset:
return
Consumer Group 1:
Partition 0: offset 13
Partition 2: offset 23](https://image.slidesharecdn.com/bs20-20230516-ksl23-babenkokutsenko-230602115643-3fd1aca0/85/The-Dark-and-Dirty-Side-of-Fixing-Uneven-Partitions-with-Olena-Babenko-Olena-Kutsenko-154-320.jpg)
The Dark and Dirty Side of Fixing Uneven Partitions with Olena Babenko & Olena Kutsenko
- 1. The dark and dirty side of fixing uneven partitions Olena Kutsenko Sr. Developer Advocate Aiven Olena Babenko Staff Software Engineer Aiven
- 2. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: It all started well…
- 3. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Recommended strategies for partitioning ➔ Select number of partitions based on how data is consumed
- 4. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Recommended strategies for partitioning ➔ Select number of partitions based on how data is consumed ➔ Select number of partitioning neither too low nor to high
- 5. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Recommended strategies for partitioning ➔ Select number of partitions based on how data is consumed ➔ Select number of partitioning neither too low nor to high ➔ Use keys with the highest cardinality
- 6. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Recommended strategies for partitioning ➔ Select number of partitions based on how data is consumed ➔ Select number of partitioning neither too low nor to high ➔ Use keys with the highest cardinality ➔ Be mindful of data distribution over time
- 7. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Recommended strategies for partitioning ➔ Select number of partitions based on how data is consumed ➔ Select number of partitioning neither too low nor to high ➔ Use keys with the highest cardinality ➔ Be mindful of data distribution over time ➔ Consider potential edge cases
- 8. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: You were pretty happy about the results
- 9. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Nothing predicted the storm
- 10. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Or so you thought
- 11. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Partition 1 47% Partition 2 34% Partition 3 7% Partition 4 4% Partition 5 4% Partition 6 4% Data balancing gone wild
- 12. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: How uneven partitions affect the system
- 13. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: How uneven partitions affect the system ➔ Brokers:
- 14. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: How uneven partitions affect the system ➔ Brokers: ◆ Heavy load on the file system -> slower brokers
- 15. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: How uneven partitions affect the system ➔ Brokers: ◆ Heavy load on the file system -> slower brokers ➔ Consumers:
- 16. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: How uneven partitions affect the system ➔ Brokers: ◆ Heavy load on the file system -> slower brokers ➔ Consumers: ◆ Increased consumer lag
- 17. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: How uneven partitions affect the system ➔ Brokers: ◆ Heavy load on the file system -> slower brokers ➔ Consumers: ◆ Increased consumer lag ◆ Consumers that are assigned to a hot partition require bigger resources
- 18. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: How uneven partitions affect the system ➔ Brokers: ◆ Heavy load on the file system -> slower brokers ➔ Consumers: ◆ Increased consumer lag ◆ Consumers that are assigned to a hot partition require bigger resources ◆ Underutilisation of resources when vertical scaling with k8s
- 19. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: How uneven partitions affect the system ➔ Brokers: ◆ Heavy load on the file system -> slower brokers ➔ Consumers: ◆ Increased consumer lag ◆ Consumers that are assigned to a hot partition require bigger resources ◆ Underutilisation of resources when vertical scaling with k8s ◆ Out-of-memory exception cycle
- 20. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: What to do now?
- 21. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: “Premature optimization is the root of all evil” Donald Knuth
- 22. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: You can’t avoid the change. Embrace the inevitable.
- 23. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Today you’ll learn ●
- 24. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Today you’ll learn ● Different recipes to deal with uneven partitioning
- 25. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Today you’ll learn ● Different recipes to deal with uneven partitioning ● From easiest 🌶
- 26. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Today you’ll learn ● Different recipes to deal with uneven partitioning ● From easiest 🌶 to more difficult 🌶🌶🌶
- 27. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: 🌶🌶🌶 The advanced techniques will help you ● Rebalance records across partitions ● Scale your topic up or down ● Be effective at disaster recovery
- 28. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Partition 1 47% Partition 2 34% Partition 3 7% Partition 4 4% Partition 5 4% Partition 6 4%
- 29. 15% 12% 13% 11% 13% 11% Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Partition 6 14% 12% Partition 7 Partition 8
- 30. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Level 1. Easy🌶
- 31. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Level 1. Easy🌶 If you don’t use keys..
- 32. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: No keys - increase the number of partitions
- 33. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: No keys - increase the number of partitions - This way you can’t scale down, but you can scale up!
- 34. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: No keys - increase the number of partitions - This way you can’t scale down, but you can scale up! - Pay attention to
- 35. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: No keys - increase the number of partitions - This way you can’t scale down, but you can scale up! - Pay attention to - Data retention period
- 36. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: No keys - increase the number of partitions - This way you can’t scale down, but you can scale up! - Pay attention to - Data retention period - Number of consumers
- 37. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: No keys - increase the number of partitions - This way you can’t scale down, but you can scale up! - Pay attention to - Data retention period - Number of consumers - Data distribution over time
- 38. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: No keys - increase the number of partitions - This way you can’t scale down, but you can scale up! - Pay attention to - Data retention period - Number of consumers - Data distribution over time - Linger_ms and batch_size for sticky partitioning
- 39. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Level 2. Moderate🌶🌶
- 40. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Level 2. Moderate🌶🌶 One or two keys are hot
- 41. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: You still can add new partitions
- 42. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: You still can add new partitions…. kinda
- 43. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: The key challenge: github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/ producer/internals/BuiltInPartitioner.java
- 44. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: The key challenge: github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/ producer/internals/BuiltInPartitioner.java
- 45. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: You will need ● Calculate which key is hot 🔥
- 46. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: You will need ● Calculate which key is hot 🔥 ● Keep the state
- 47. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: You will need ● Calculate which key is hot 🔥 ● Keep the state ● Not mess up old keys
- 48. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: You will need ● Calculate which key is hot 🔥 ● Keep the state ● Not mess up old keys ● Use custom partitioner
- 49. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Example public static int partitionForKey(final byte[] serializedKey, final int numPartitions) { if (serializedKey == "bananas🍌🍌") { ... do the dirty magic here ... } else { return Utils.toPositive(Utils.murmur2(serializedKey)) % (numPartitions - 1); } }
- 50. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Example public static int partitionForKey(final byte[] serializedKey, final int numPartitions) { if (serializedKey == "bananas🍌🍌") { ... do the dirty magic here ... } else { return Utils.toPositive(Utils.murmur2(serializedKey)) % (numPartitions - 1); } }
- 51. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Example public static int partitionForKey(final byte[] serializedKey, final int numPartitions) { if (serializedKey == "bananas🍌🍌") { ... do the dirty magic here ... } else { return Utils.toPositive(Utils.murmur2(serializedKey)) % (numPartitions - 1); } }
- 52. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Level 3. Getting hot🌶🌶🌶
- 53. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Level 3. Getting hot🌶🌶🌶 Time to migrate to a new topic
- 54. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: The time will come… when you need to re-create the topic
- 55. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: The time will come… when you need to re-create the topic ➔ Rebalance records across partitions
- 56. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: The time will come… when you need to re-create the topic ➔ Rebalance records across partitions ➔ Scale your topic up or down
- 57. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: The time will come… when you need to re-create the topic ➔ Rebalance records across partitions ➔ Scale your topic up or down ➔ To do disaster recovery
- 58. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Let’s re-create the topic and MIGRATE!
- 59. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Three main steps
- 60. … Producers Consumers P1 P0 Old topic 0 6 12 🗝 🗝 1 7 13
- 61. … Producers Consumers P1 P0 Old topic 0 6 12 🗝 🗝 1 7 13 P1 P2 P0 P3 0 12 🗝 1 13 🗝 2 14 3 15 🗝 🗝 New topic
- 62. … … Producers Consumers P1 P0 P1 P2 P0 P3 Old topic New topic 0 6 12 🗝 🗝 0 12 🗝 1 13 🗝 1 7 13 2 14 3 15 🗝 🗝
- 63. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Our goals when migrating 1. Keep downtime to bare minimum 2. No duplicates
- 64. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Partition 1 47% Partition 2 34% Partition 3 7% Partition 4 4% Partition 5 4% Partition 6 4%
- 65. 15% 12% 13% 11% 13% 11% Partition 1 Partition 2 Partition 3 Partition 4 Partition 5 Partition 6 14% 12% Partition 7 Partition 8
- 66. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Two options
- 67. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Two options (in reality way more, but similar in essence)
- 68. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Option 1: Stop the world producers
- 69. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Option 1: Stop the world producers Sharp cut
- 70. … Producers Consumers P1 P0 Old topic 0 6 12 🗝 🗝 1 7 13
- 71. … Producers Consumers P1 P0 Old topic 0 6 12 🗝 🗝 1 7 13
- 72. … Producers Consumers P1 P0 Old topic 0 6 12 🗝 🗝 1 7 13
- 73. … Producers Consumers P1 P0 Old topic 0 6 12 🗝 🗝 1 7 13 P1 P2 P0 P3 New topic 0 12 🗝 1 13 🗝 2 14 3 15 🗝 🗝
- 74. … Producers Consumers P1 P0 Old topic 0 6 12 🗝 🗝 1 7 13 P1 P2 P0 P3 New topic 0 12 🗝 1 13 🗝 2 14 3 15 🗝 🗝
- 75. … Producers Consumers P1 P0 0 6 12 🗝 🗝 1 7 13 P1 P2 P0 P3 New topic 0 12 🗝 1 13 🗝 2 14 3 15 🗝 🗝 Old topic
- 76. … Consumers P1 P0 0 6 12 🗝 🗝 1 7 13 P1 P2 P0 P3 New topic 0 12 🗝 1 13 🗝 2 14 3 15 🗝 🗝 Producers Old topic
- 77. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Advantages ➔ No skipped messages ➔ Prevention of duplicates ➔ No need for extra compute to replicate data from old to new topic
- 78. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Limitations ➔ Downtime ➔ Difficult to test new setup and challenging to roll back ➔ Limited time window for migration ➔ Need for seamless collaboration among teams ➔ All-or-nothing migration
- 79. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Our goals when migrating 1. Keep downtime to bare minimum 2. No duplicates
- 80. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Option 2: Gradual switch relying on replicated data
- 81. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: ABOVE - Olena K BELOW - Olena B
- 82. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Time for plan B
- 83. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Time for plan B with Olena B
- 84. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Strategy
- 85. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 1
- 86. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 2
- 87. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 3
- 88. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 3
- 89. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 3
- 90. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 3
- 91. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 4
- 92. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 4
- 93. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 5
- 94. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 1 New topic creation
- 95. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 1
- 96. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Partition 1 47% Partition 2 34% Partition 3 7% Partition 4 4% Partition 5 4% Partition 6 4%
- 97. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Partition 1 44% Partition 2 30% Partition 3 6% Partition 4 3% Partition 5 3% Partition 6 3% Partition 7 3% Partition 8 3%
- 98. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Risks Had to redo whole process because of too few/many partitions
- 99. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 2 Fast and reliable data pump
- 100. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 2
- 101. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Data pump application requirements - Simple - Fast. - Reliable Kafka Streams Java
- 102. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Risks - Require too much resources if not simple enough - Cannot keep up if it is too complicated - Data losses if application is not reliable - Data loss or duplicates because records from from different partitions get shuffled WARNING. Records/keys almost certainly will be mixed.
- 103. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: New partitions have mix of data from old partitions
- 104. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: New partitions have mix of data from old partitions
- 105. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: New partitions have mix of data from old partitions
- 106. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: New partitions have mix of data from old partitions
- 107. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: New partitions have mix of data from old partitions
- 108. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: New partitions have mix of data from old partitions
- 109. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: New partitions have mix of data from old partitions
- 110. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: New partitions have mix of data from old partitions
- 111. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: New partitions have mix of data from old partitions
- 112. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: New partitions have mix of data from old partitions
- 113. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: New partitions have mix of data from old partitions
- 114. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: New partitions have mix of data from old partitions
- 115. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: New partitions have mix of data from old partitions
- 116. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: New partitions have mix of data from old partitions
- 117. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: New partitions have mix of data from old partitions
- 118. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: New partitions have mix of data from old partitions
- 119. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Out of order events
- 120. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Out of order events
- 121. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Out of order events
- 122. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Out of order events
- 123. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Out of order events
- 124. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Out of order events
- 125. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Out of order events
- 126. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Out of order events
- 127. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Out of order events
- 128. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Out of order events If consumers had stopped when order is not correct. - Read some records one more time OR - Skip some records
- 129. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Out of order events
- 130. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Out of order events Be careful during data pump catch up and if you use big batches to read data
- 131. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Out of order events Old topic timestamps from metadata could be used to preserve chronological order
- 132. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 3 Gradual consumer switch.
- 133. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 3
- 134. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Risks - Spikes - Too long downtime for consumers - Data loss or duplicates
- 135. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 3
- 136. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Consumer groups translations
- 137. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Simple Consumer Group Translation
- 138. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Simple Consumer Group Translation
- 139. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Simple Consumer Group Translation Old Consumer Group: Partition 0: offset 13 Partition 2: offset 23 Last consumed event Partition 0: offset 12 Partition 2: offset 22
- 140. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Simple Consumer Group Translation Timestamps: Partition 0: 07:01:04 Partition 2: 07:01:03 Earliest timestamp: 07:01:03
- 141. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Simple Consumer Group Translation
- 142. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Risks - Spikes - Too long downtime for consumers - Data loss or duplicates
- 143. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Streaming Consumer Group Translations
- 144. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Streaming Consumer Group Translation
- 145. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Streaming Consumer Group Translation
- 146. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Streaming Consumer Group Translation
- 147. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Streaming Consumer Group Translation
- 148. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Streaming Consumer Group Translation
- 149. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Risks - Spikes - Too long downtime for consumers - Data loss or duplicates
- 150. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Offset Translations
- 151. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Offset Translation
- 152. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Offset Translation
- 153. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Offset Translation Consumer Group 1: Partition 0: offset 13 Partition 2: offset 23
- 154. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Offset Translation For r in records: P = r.metadata.old_partition If offsets[P] <= r.metadata.offset: return Consumer Group 1: Partition 0: offset 13 Partition 2: offset 23
- 155. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Mirror Maker offset translation Data pump -> MirrorSourceTask Old + New records metadata -> Records in Offset Sync topics Offset translation -> MirrorCheckpointTask Problems: - Main usecase data transfer between 2 clusters, not a same - Till version 3.3 offset translation by measuring the 'distance' between the MM2 offset sync and the upstream consumer group, and then assuming that the same distance applies in the downstream topic.
- 156. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Risks - Spikes - Too long downtime for consumers - Data loss or duplicates - Poor offsets estimations - Bad timing for offsets translation
- 157. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Bad timing for offset translations
- 158. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Bad timing for Offset Translation
- 159. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Bad timing for Offset Translation Either Start from 32: duplicate B1, B2 OR Start from 35: A2 is lost
- 160. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: “Bounded” stream
- 161. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Gradual consumer switch Earliest offset Duplicates guaranteed Consumer’s earliest timestamp High probability of duplicates Offset translation A few duplicates Offset translation + “late events” tracking Almost no duplicates Offset translation + “late events” tracking + “Bounded” stream approach Rare/no duplicates
- 162. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Gradual consumer switch
- 163. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: More topics to talk about - Apache Mirror Maker implementation details - Stateless vs Stateful consumers - Idempotence - Changing schemas - New key selection strategy
- 164. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 4 Gradual producers switch
- 165. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Step 4
- 166. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Risks - Data loss or duplicates if data pump is not fast enough
- 167. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: To summarize it all
- 168. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Key learnings ● No keys - add partitions 🌶 ● A few hot keys - you still can add partitions 🌶🌶 ● Workarounds are not sufficient? - Migrate the topic 🌶🌶🌶
- 169. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Migrate the topic 🌶🌶🌶 ● Sharp cut - stop the producers first ○ Exactly once delivery ○ Expect the downtime ○ All-or-nothing migration ● Generic gradual switch ○ Minimal downtime ○ Possibility to test before switching ○ Switch consumer groups gradually ○ Minimize chance of duplicates
- 170. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko:
- 171. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko:
- 172. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Olena Kutsenko Sr. Developer Advocate Olena Babenko Staff Software Engineer
- 173. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Olena Kutsenko Sr. Developer Advocate Olena Babenko Staff Software Engineer
- 174. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: Olena Kutsenko Sr. Developer Advocate Olena Babenko Staff Software Engineer The trusted open source data platform for everyone
- 175. olena@aiven.io @OlenaKutsenko aiven.io Olena Babenko: #G8 The trusted open source data platform for everyone