🐲 Here be Stacktraces — Flink SQL for Non-Java Developers
0 likes187 views
The document discusses Apache Flink, its capabilities, and how to use Flink SQL for non-Java developers. It covers the system's history, its components, SQL syntax, and practical examples of queries and table creation. The document emphasizes the importance of understanding dependencies and configurations for effectively utilizing Flink SQL.
![@rmoff / #kafkasum
m
i
t
@decodableco
Stateful and Stateless computations
• Filtering
• Joining
• Transfor
m
i
ng
• Aggregations
• Pattern matching
• …and a whole lot more
SELECT * FROM myStream WHERE foo=42
SELECT a.*, b.* FROM myStream a
INNER JOIN myLookup b ON a.id=b.foo_id
SELECT cost * tax_rate AS total_cost
FROM myStream
SELECT SU
M
(order_value) AS total_order_values
FROM orders
SELECT *
FROM myStream
MATCH_RECOGNIZE (
PART
I
T
I
ON BY id
ORDER BY user_action_t
i
m
e
[…]](https://image.slidesharecdn.com/bs16-20240319-ksl24-decodable-moffattrobin-240402153333-2848c8b5/85/Here-be-Stacktraces-Flink-SQL-for-Non-Java-Developers-12-320.jpg)
![@rmoff / #kafkasum
m
i
t
@decodableco
Runt
i
m
e Mode
S
E
T
'execution.runt
i
m
e-mode' = 'strea
m
i
ng';
• strea
m
i
ng [default]
• batch](https://image.slidesharecdn.com/bs16-20240319-ksl24-decodable-moffattrobin-240402153333-2848c8b5/85/Here-be-Stacktraces-Flink-SQL-for-Non-Java-Developers-27-320.jpg)
![@rmoff / #kafkasum
m
i
t
@decodableco
Result Mode
S
E
T
'sql-client.execution.result-mode' = 'table';
• table [default]
• changelog
• tableau](https://image.slidesharecdn.com/bs16-20240319-ksl24-decodable-moffattrobin-240402153333-2848c8b5/85/Here-be-Stacktraces-Flink-SQL-for-Non-Java-Developers-28-320.jpg)
🐲 Here be Stacktraces — Flink SQL for Non-Java Developers
- 1. Robin Moffatt, Principal DevEx Engineer @ Decodable @rmoff / 19 Mar 2024 / #kafkasum m i t 🐲 Here be Dragons Stacktraces Flink SQL for Non-Java Developers
- 2. @rmoff / #kafkasum m i t @decodableco Actual footage of a SQL Developer looking at Apache Flink for the first t i m e
- 4. @rmoff / #kafkasum m i t @decodableco What Is Apache Flink?
- 5. @rmoff / #kafkasum m i t @decodableco A Brief History of Flink
- 8. @rmoff / #kafkasum m i t @decodableco • Started life as a research project in 2 0 1 1, called Stratosphere. • This was the t i m e of MapReduce. Java and Scala were the only way to do this. Back in the t i m e of dinosaurs Hadoop
- 9. @rmoff / #kafkasum m i t @decodableco Flink is a big project • Flink • Stateful Functions • M L • Kubernetes Operator • CDC Connector • Pa i m on (incubating)
- 11. @rmoff / #kafkasum m i t @decodableco Connect to Lots of Source and Target Systems DynamoDB Object stores Kinesis Firehose Kinesis JDBC CDC
- 12. @rmoff / #kafkasum m i t @decodableco Stateful and Stateless computations • Filtering • Joining • Transfor m i ng • Aggregations • Pattern matching • …and a whole lot more SELECT * FROM myStream WHERE foo=42 SELECT a.*, b.* FROM myStream a INNER JOIN myLookup b ON a.id=b.foo_id SELECT cost * tax_rate AS total_cost FROM myStream SELECT SU M (order_value) AS total_order_values FROM orders SELECT * FROM myStream MATCH_RECOGNIZE ( PART I T I ON BY id ORDER BY user_action_t i m e […]
- 13. @rmoff / #kafkasum m i t @decodableco Batch and Strea m i ng Bounded and Unbounded
- 14. @rmoff / #kafkasum m i t @decodableco How Does Flink Work?
- 15. @rmoff / #kafkasum m i t @decodableco << magic 🪄 🧙 >>
- 17. @rmoff / #kafkasum m i t @decodableco https: // nightlies.apache.org/flink/flink-docs-master/docs/deployment/overview/ Flink's SQL Engine: Let's Open the Engine Room! 🗣 T i m o Walther 📆 Tuesday ⏰ 5:30pm 🗺 Breakout Room 2
- 18. @rmoff / #kafkasum m i t @decodableco Running Flink Works on my machine… $ ./bin/start-cluster.sh Starting cluster. Starting standalonesession daemon on host asgard08. Starting taskexecutor daemon on host asgard08. $ jps -l 14656 org.apache.flink.runt i m e.taskexecutor. T askManagerRunner 14379 org.apache.flink.runt i m e.entrypoint.StandaloneSessionClusterEntrypoint
- 20. @rmoff / #kafkasum m i t @decodableco It's not just Java • PyFlink • added in 1.9. 0 in 2 0 1 9 • Flink SQL • Added in 1.5. 0 in 2 0 1 8
- 22. @rmoff / #kafkasum m i t @decodableco • Built on Apache Calcite • Common Table Expression (C T E ) ( W I T H) • Set-based operations • Joins • Aggregations • And lots more… SQL Language Support
- 23. @rmoff / #kafkasum m i t @decodableco Running Flink SQL • SQL Client • SQL Gateway • RE S T API • Hive • JDBC Driver • From Java or Python
- 24. @rmoff / #kafkasum m i t @decodableco ▒▓██▓██▒ ▓████▒▒█▓▒▓███▓▒ ▓███▓░░ ▒▒▒▓██▒ ▒ ░██▒ ▒▒▓▓█▓▓▒░ ▒████ ██▒ ░▒▓███▒ ▒█▒█▒ ░▓█ ███ ▓░▒██ ▓█ ▒▒▒▒▒▓██▓░▒░▓▓█ █░ █ ▒▒░ ███▓▓█ ▒█▒▒▒ ████░ ▒▓█▓ ██▒▒▒ ▓███▒ ░▒█▓▓██ ▓█▒ ▓█▒▓██▓ ░█░ ▓░▒▓████▒ ██ ▒█ █▓░▒█▒░▒█▒ ███▓░██▓ ▓█ █ █▓ ▒▓█▓▓█▒ ░██▓ ░█░ █ █▒ ▒█████▓▒ ██▓░▒ ███░ ░ █░ ▓ ░█ █████▒░░ ░█░▓ ▓░ ██▓█ ▒▒▓▒ ▓███████▓░ ▒█▒ ▒▓ ▓██▓ ▒██▓ ▓█ █▓█ ░▒█████▓▓▒░ ██▒▒ █ ▒ ▓█▒ ▓█▓ ▓█ ██▓ ░▓▓▓▓▓▓▓▒ ▒██▓ ░█▒ ▓█ █ ▓███▓▒░ ░▓▓▓███▓ ░▒░ ▓█ ██▓ ██▒ ░▒▓▓███▓▓▓▓▓██████▓▒ ▓███ █ ▓███▒ ███ ░▓▓▒░░ ░▓████▓░ ░▒▓▒ █▓ █▓▒▒▓▓██ ░▒▒░░░▒▒▒▒▓██▓░ █▓ ██ ▓░▒█ ▓▓▓▓▒░░ ▒█▓ ▒▓▓██▓ ▓▒ ▒▒▓ ▓█▓ ▓▒█ █▓░ ░▒▓▓██▒ ░▓█▒ ▒▒▒░▒▒▓█████▒ ██░ ▓█▒█▒ ▒▓▓▒ ▓█ █░ ░░░░ ░█▒ ▓█ ▒█▓ ░ █░ ▒█ █▓ █▓ ██ █░ ▓▓ ▒█▓▓▓▒█░ █▓ ░▓██░ ▓▒ ▓█▓▒░░░▒▓█░ ▒█ ██ ▓█▓░ ▒ ░▒█▒██▒ ▓▓ ▓█▒ ▒█▓▒░ ▒▒ █▒█▓▒▒░░▒██ ░██▒ ▒▓▓▒ ▓██▓▒█▒ ░▓▓▓▓▒█▓ ░▓██▒ ▓░ ▒█▓█ ░░▒▒▒ ▒▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░▓▓ ▓░▒█░ ______ _ _ _ _____ ____ _ _____ _ _ _ BETA | ____| (_) | | / ____|/ __ | | / ____| (_) | | | |__ | |_ _ __ | | __ | (___ | | | | | | | | |_ ___ _ __ | |_ | __| | | | '_ | |/ / ___ | | | | | | | | | |/ _ '_ | __| | | | | | | | | < ____) | |__| | |____ | |____| | | __/ | | | |_ |_| |_|_|_| |_|_|_ |_____/ __________| _____|_|_|___|_| |_|__| Flink SQL Client $ ./bin/sql-client.sh Welcome! Enter 'HELP;' to list all available commands. 'QU I T ;' to exit. Command history file path: /opt/ flink/.flink-sql-history Flink SQL>
- 25. @decodableco @rmoff / #kafkasum m i t D E M O https: // github.com/decodableco/examples/kafka-iceberg
- 26. @rmoff / #kafkasum m i t @decodableco A Few Useful Settings
- 27. @rmoff / #kafkasum m i t @decodableco Runt i m e Mode S E T 'execution.runt i m e-mode' = 'strea m i ng'; • strea m i ng [default] • batch
- 28. @rmoff / #kafkasum m i t @decodableco Result Mode S E T 'sql-client.execution.result-mode' = 'table'; • table [default] • changelog • tableau
- 29. @rmoff / #kafkasum m i t @decodableco Colour Scheme S E T 'sql-client.display.color-schema' = 'Chester'; • Because why not?!
- 30. @rmoff / #kafkasum m i t @decodableco Changing the defaults Setting up a SQL Client initialisation file • Create a SQL file: $ cat init.sql S E T 'execution.runt i m e-mode' = 'batch'; S E T 'sql-client.execution.result-mode' = 'tableau'; • Launch SQL Client w i th the -i flag and pass the file as a parameter: ./bin/sql-client.sh -i init.sql
- 31. @rmoff / #kafkasum m i t @decodableco Sub m i tting SQL as a job • SQL Client $ ./bin/sql-client.sh --file ~/my_query.sql • SQL Gateway curl --location 'localhost:8083/sessions/42/statements' --header 'Content-Type: application/json' --header 'Accept: application/json' --data '{ "statement": "SELECT * FROM foo;" }' • Application mode support: FLIP-316: Support application mode for SQL Gateway
- 32. @rmoff / #kafkasum m i t @decodableco Some of the Gnarly Stuff
- 33. @rmoff / #kafkasum m i t @decodableco • For each connector, format, and catalog you need to install dependencies. • All of these are available as JARs (Java ARchive) The Joy of JARs
- 34. @rmoff / #kafkasum m i t @decodableco This m i ght jar a bit… Could not execute SQL statement. Reason: java.lang.ClassNotFoundException Could not find any factory for identifier 'hive' that i m plements 'org.apache.flink.table.factories.CatalogF actory' in the classpath. org.apache.flink.core.fs.UnsupportedFileSy ste m S chemeException: Could not find a file system i m plementation for scheme 's3'
- 35. @rmoff / #kafkasum m i t @decodableco Finding JARs • Usually the docs w i ll tell you which JAR you need. • JARs are very specific to the versions of the tools that you're using.
- 39. @rmoff / #kafkasum m i t @decodableco $ tree /opt/flink/lib ├── aws │ ├── aws-java-sdk-bundle-1.12.648.jar │ └── hadoop-aws-3.3.4.jar ├── flink-cep-1.18.1.jar ├── flink-parquet_2.12-1.18.1.jar ├── flink-table-runt i m e-1.18.1.jar ├── hadoop │ ├── commons-configuration2-2.1.1.jar │ ├── commons-logging-1.1.3.jar │ ├── hadoop-auth-3.3.4.jar ├── hive │ └── flink-sql-connector-hive-3.1.3_2.12-1.18.1.jar ├── iceberg │ └── iceberg-flink-runt i m e-1.18-1.5. 0 .jar ├── kafka │ └── flink-sql-connector-kafka-3.1. 0 -1.18.jar
- 40. @decodableco @rmoff / #kafkasum m i t Don't forget to restart!
- 41. @rmoff / #kafkasum m i t @decodableco Tables, Connectors, and Catalogs
- 42. @rmoff / #kafkasum m i t @decodableco Tables CREA T E TABLE t_k_orders ( orderid S T RING, customerid S T RING, ordernumber IN T , product S T RING, discountpercent INT ) W I T H ( 'connector' = 'kafka', 'topic' = 'orders', 'properties.bootstrap.servers' = 'broker:29092', 'scan.startup. m ode' = 'earliest-offset', 'format' = 'json' );
- 43. @rmoff / #kafkasum m i t @decodableco • The data and information about the data was all stored in the database • Information Schema • System Catalog • Data Dictionary Views This used to be s i m ple
- 44. @decodableco @rmoff / #kafkasum m i t Now it's not so s i m ple
- 45. @rmoff / #kafkasum m i t @decodableco Flink Catalogs
- 46. @rmoff / #kafkasum m i t @decodableco Flink Catalogs
- 47. @rmoff / #kafkasum m i t @decodableco Flink Catalogs
- 48. @rmoff / #kafkasum m i t @decodableco Flink Catalogs
- 49. @rmoff / #kafkasum m i t @decodableco Flink Catalogs
- 50. @rmoff / #kafkasum m i t @decodableco Flink Catalogs CREA T E CATALOG c_hive W I T H ( 'type' = 'hive', 'hive-conf-dir' = './conf');
- 51. @rmoff / #kafkasum m i t @decodableco Flink Catalogs table.catalog-store.kind: file table.catalog-store.file.path: ./conf/catalogs
- 52. @decodableco @rmoff / #kafkasum m i t D E M O https: // github.com/decodableco/examples/kafka-iceberg
- 53. @rmoff / #kafkasum m i t @decodableco In Conclusion…
- 54. @rmoff / #kafkasum m i t @decodableco Flink SQL is Fun! But there's a bit of a learning curve • Run ad-hoc queries w i th the SQL Client • Understand JAR dependencies for connectors, catalogs, formats, etc • Don't be put off by the docs - there i s SQL content there if you look hard enough
- 56. @decodableco @rmoff / #kafkasum m i t # E OF @rmoff / 19 Mar 2024 / #kafkasum m i t