KSQL in Practice (Almog Gavra, Confluent) Kafka Summit London 2019
0 likes1,706 views
The document provides a comprehensive overview of ksql, a streaming SQL engine for Apache Kafka, covering its introduction, development, deployment, and operation. Key aspects include performance, common mistakes, and fraud detection use cases, along with details on creating streams and tables. It also addresses deployment options, scalability, and query handling practices, such as continuous queries and data processing features.
![18
Import Data Create Stream/Table Write KSQL Queries Verify Output
* Feature Coming in CP 5.3
ksql-test-framework*
{
"statements": [
"CREATE STREAM TEST (source VARCHAR) WITH (kafka_topic='test_topic');",
"CREATE STREAM O AS SELECT CONCAT('prefix-', source) AS C FROM TEST;"],
"inputs": [
{"topic": "test_topic", "value": {"source": "s1"}},
{"topic": "test_topic", "value": {"source": "s2"}],
"outputs": [
{"topic": "O", "value": {"C":"prefix-s1"}},
{"topic": "O", "value": {“C":"prefix-s2"}}]
}](https://image.slidesharecdn.com/almoggavraconfluentksqlinpractice-190522013711/85/KSQL-in-Practice-Almog-Gavra-Confluent-Kafka-Summit-London-2019-34-320.jpg)
![18
Import Data Create Stream/Table Write KSQL Queries Verify Output
* Feature Coming in CP 5.3
ksql-test-framework*
{
"statements": [
"CREATE STREAM TEST (source VARCHAR) WITH (kafka_topic='test_topic');",
"CREATE STREAM O AS SELECT CONCAT('prefix-', source) AS C FROM TEST;"],
"inputs": [
{"topic": "test_topic", "value": {"source": "s1"}},
{"topic": "test_topic", "value": {"source": "s2"}],
"outputs": [
{"topic": "O", "value": {"C":"prefix-s1"}},
{"topic": "O", "value": {“C":"prefix-s2"}}]
}](https://image.slidesharecdn.com/almoggavraconfluentksqlinpractice-190522013711/85/KSQL-in-Practice-Almog-Gavra-Confluent-Kafka-Summit-London-2019-35-320.jpg)
KSQL in Practice (Almog Gavra, Confluent) Kafka Summit London 2019
- 1. KSQL in Practice Almog Gavra Engineer, KSQL
- 2. 2 ● Intro ● Develop ● Deploy ● Operate ● Common Mistakes ● Q&A
- 3. 3 Why KSQL?
- 4. 3 Performance and Robustness from Apache Kafka® Why KSQL?
- 5. 3 Performance and Robustness from Apache Kafka® Simplified Stream Processing Why KSQL?
- 6. 3 Performance and Robustness from Apache Kafka® ksql>Familiar & Expressive SQL-Like Language Simplified Stream Processing Why KSQL?
- 7. 4 SELECT * FROM authorizations WHERE card_number = 123; User submits SQL
- 8. 4 SELECT * FROM authorizations WHERE card_number = 123; User submits SQL KSQL builds KafkaStreams topology
- 9. 4 SELECT * FROM authorizations WHERE card_number = 123; User submits SQL KSQL builds KafkaStreams topology Kafka Streams KafkaStreams executes the topology
- 10. 5 Fraud Detection via KSQL “find all card numbers that had more than 3 authorization attempts within 5 seconds”
- 11. 5 Fraud Detection via KSQL CREATE TABLE possible_frauds AS SELECT card_number, count(*) FROM authorizations GROUP BY card_number HAVING count(*) > 3; “find all card numbers that had more than 3 authorization attempts within 5 seconds”
- 12. 5 Fraud Detection via KSQL CREATE TABLE possible_frauds AS SELECT card_number, count(*) FROM authorizations GROUP BY card_number HAVING count(*) > 3; WINDOW TUMBLING (SIZE 5 SECONDS) “find all card numbers that had more than 3 authorization attempts within 5 seconds”
- 13. 6 ● Intro ● Develop ● Deploy ● Operate ● Common Mistakes ● Q&A
- 14. 7 but first…
- 16. 8 T SA X B Y CAA XX BB YY CC topic A key value
- 17. 8 T SA X B Y C A A XX BB YY CC topic A key value
- 18. 8 T SA X B Y C A A X X BB YY CC topic A key value
- 19. 8 T SA X B Y C A A X X B B YY CC topic A key value
- 20. 8 T SA X B Y C A A X X B BY Y CC topic A key value
- 21. 8 T SA X B Y C A A X X B BY Y C C topic A key value
- 22. 8 T SA X B Y C A A X X B BY Y C C topic ‣ a table is the current state of your data ‣ the topic is a changelog A key value
- 23. 8 T SA X B Y C A A X X B BY Y C C topic ‣ a table is the current state of your data ‣ the topic is a changelog ‣ a stream is the historical state of your data ‣ the topic is a sequence A key value
- 24. 9 Import Data Create Stream/Table Write KSQL Queries Verify Output Development Lifecycle
- 25. 10 Import Data Create Stream/Table Write KSQL Queries Verify Output Kafka Connect INSERT INTO foo VALUES * * Feature Coming in CP 5.3 ./ksql-datagen
- 26. 11 Import Data Create Stream/Table Write KSQL Queries Verify Output CREATE TABLE/STREAM name (schema …) WITH (kafka_topic=“topic”); DDL
- 27. 11 Import Data Create Stream/Table Write KSQL Queries Verify Output CREATE TABLE/STREAM name (schema …) WITH (kafka_topic=“topic”); DDL CREATE TABLE/STREAM name (schema …) AS SELECT * FROM source; DML
- 28. 12 Import Data Create Stream/Table Write KSQL Queries Verify Output { REST }
- 29. 13 Import Data Create Stream/Table Write KSQL Queries Verify Output UDF UDAF
- 30. 14 Import Data Create Stream/Table Write KSQL Queries Verify Output UDF SELECT amount, OBFUSCATE(card_number) AS id FROM AUTHORIZATIONS;
- 31. 15 Import Data Create Stream/Table Write KSQL Queries Verify Output UDAF SELECT card_number, SUM(cost) AS total_cost FROM AUTHORIZATIONS WINDOW TUMBLING (SIZE 7 DAYS) GROUP BY card_number;
- 32. 16 Import Data Create Stream/Table Write KSQL Queries Verify Output UDAF SELECT card_number, SUM(cost) AS total_cost FROM AUTHORIZATIONS WINDOW TUMBLING (SIZE 7 DAYS) GROUP BY card_number; TS (123, $10) (123, $20) (456, $40) (123, $5) (123_w1, $30) (456_w1, $40) (123_w2, $5)
- 33. 17 Import Data Create Stream/Table Write KSQL Queries Verify Output SELECT * FROM stream LIMIT 3 ksql-test-framework* * Feature Coming in CP 5.3
- 34. 18 Import Data Create Stream/Table Write KSQL Queries Verify Output * Feature Coming in CP 5.3 ksql-test-framework* { "statements": [ "CREATE STREAM TEST (source VARCHAR) WITH (kafka_topic='test_topic');", "CREATE STREAM O AS SELECT CONCAT('prefix-', source) AS C FROM TEST;"], "inputs": [ {"topic": "test_topic", "value": {"source": "s1"}}, {"topic": "test_topic", "value": {"source": "s2"}], "outputs": [ {"topic": "O", "value": {"C":"prefix-s1"}}, {"topic": "O", "value": {“C":"prefix-s2"}}] }
- 35. 18 Import Data Create Stream/Table Write KSQL Queries Verify Output * Feature Coming in CP 5.3 ksql-test-framework* { "statements": [ "CREATE STREAM TEST (source VARCHAR) WITH (kafka_topic='test_topic');", "CREATE STREAM O AS SELECT CONCAT('prefix-', source) AS C FROM TEST;"], "inputs": [ {"topic": "test_topic", "value": {"source": "s1"}}, {"topic": "test_topic", "value": {"source": "s2"}], "outputs": [ {"topic": "O", "value": {"C":"prefix-s1"}}, {"topic": "O", "value": {“C":"prefix-s2"}}] }
- 36. 19 ● Intro ● Develop ● Deploy ● Operate ● Common Mistakes ● Q&A
- 39. 20 Interactive Mode Headless Mode Deployment Options
- 40. 21 Interactive Mode Headless Mode
- 41. 21 Interactive Mode Headless Mode
- 42. 22 KSQL scales relative to partitions in Kafka
- 43. 22 KSQL scales relative to partitions in Kafka … and rebalances if a replica is lost
- 44. 22 KSQL scales relative to partitions in Kafka … and rebalances if a replica is lost✓ tip: initially over partitioning your input topics can help scale!
- 45. 23
- 47. 24
- 53. 28 group queries by use- case using service.id Recommended Deployment Topology
- 54. 29 group queries by use- case using service.id scale by increasing replicas Recommended Deployment Topology
- 55. 29 group queries by use- case using service.id scale by increasing replicas queries auto-assign application.id Recommended Deployment Topology
- 56. 30 Recommended Deployment Topology group queries by use- case using service.id queries auto-assign application.id
- 60. 34 filters projections joins aggregates 100 EPS 100 EPS 50 EPS 25 EPS * assume that your max throughput is 100 EPS (it is likely you can support orders of magnitude more) *
- 62. 35 joins aggregates S T S S T T see: https://docs.confluent.io/current/streams/developer-guide/dsl-api.html#stateful-transformations
- 63. 35 joins aggregates S T S S T T see: https://docs.confluent.io/current/streams/developer-guide/dsl-api.html#stateful-transformations
- 64. 36 ● Intro ● Develop ● Deploy ● Operate ● Common Mistakes ● Q&A
- 70. 38 processing log KSQL_PROCESSING_LOG ✓ tip: the best way to debug streaming is by streaming
- 71. 39
- 73. 39 JMX Interceptors error-rate: 0.0 num-persistent-queries: 2.0 num-active-queries: 2.0 messages-consumed-per-sec: 193024.78294586178 messages-produced-per-sec: 193025.4730374501 messages-consumed-max: 103397.81191436431
- 74. 40
- 75. 40
- 76. 40
- 80. 42 ● Intro ● Develop ● Deploy ● Operate ● Common Mistakes ● Q&A
- 83. 44
- 84. 44 new queries read from most recent by default
- 85. 44 new queries read from most recent by default SET ‘auto.offset.reset’ = ‘earliest’
- 86. 45 joins may cause repartition A B B D C B A C B A
- 87. 45 joins may cause repartition A B B D C B A C B A A B BB A B D CC
- 88. 46 repartition does not preserve order across partitions…
- 89. 46 repartition does not preserve order across partitions… T
- 90. 46 repartition does not preserve order across partitions… T so tables cannot be rekeyed
- 91. 47
- 92. 47 crashes replay (live query) state
- 93. 48 ksql>
- 96. 50 ● Intro ● Develop ● Deploy ● Operate ● Common Mistakes ● Q&A