Flink sql for continuous sql etl apps & Apache NiFi devops
2 likes658 views
Timothy Spann, a Principal DataFlow Field Engineer at Cloudera, gave a presentation about Apache Flink SQL for continuous SQL/ETL/applications and Apache NiFi for DevOps. The presentation included demos of building real-time streaming pipelines with Flink and using the NiFi CLI, REST API, and NiPyAPI for NiFi DevOps. Upcoming events were also announced.
Flink sql for continuous sql etl apps & Apache NiFi devops
- 1. Talk 1: Flink SQL For Continuous SQL/ETL/Apps Talk 2: Apache NiFi DevOps Timothy Spann - Principal DataFlow Field Engineer 25-March-2021 https://www.meetup.com/futureofdata-princeton/ @PaasDev
- 2. © 2021 Cloudera, Inc. All rights reserved. 2 Upcoming Events ● 7/April/2021 - CDF - Ask the Experts ● 15/April/2021 - Real-time Streaming Pipelines with FLaNK DataConLA ● 21/April/2021 - Emerging Tech Day ● 22/April/2021- Demo Jam NiFi ● 27/April/2021 - Developer Week Europe ● 06/May/2021 - Continuous SQL with SQL Stream Builder ● https://www.meetup.com/pro/futureofdata/ ● https://www.linkedin.com/pulse/2021-schedule-tim-spann/ ● https://www.meetup.com/futureofdata-newyork
- 3. © 2021 Cloudera, Inc. All rights reserved. 3 Welcome to Future of Data - Virtual - 25/March/2021 @PaasDev https://www.meetup.com/futureofdata-princeton/ https://www.meetup.com/futureofdata-newyork/ https://www.meetup.com/futureofdata-philadelphia/ From Big Data to AI to Streaming to Containers to Cloud to Analytics to Cloud Storage to Fast Data to Machine Learning to Microservices to ...
- 4. © 2021 Cloudera, Inc. All rights reserved. 4
- 5. © 2021 Cloudera, Inc. All rights reserved. 5 https://github.com/tspannhw https://www.datainmotion.dev/
- 6. © 2021 Cloudera, Inc. All rights reserved. 6
- 7. © 2021 Cloudera, Inc. All rights reserved. 7 Simplifying the User Experience
- 8. © 2021 Cloudera, Inc. All rights reserved. 8 REST Data Collect Transform Demo Scenario Scalable collection Publish / Analyze Scalable transformation Alerting Analyze CLOUD DATA STORES Event Stream CLOUDERA DATA PLATFORM
- 9. © 2021 Cloudera, Inc. All rights reserved. 9 End to End Demo Pipeline Enterprise sources Weather Errors Aggregates Alerts Stocks ETL Analytics Streaming SQL Clickstream Market data Machine logs Social
- 10. © 2021 Cloudera, Inc. All rights reserved. 10 Weather Streaming Pipeline Weather Weather Climate Aggregates Sensors SQL Analytics Sources Pollution
- 11. © 2021 Cloudera, Inc. All rights reserved. 11 https://github.com/tspannhw/SmartStocks
- 12. © 2021 Cloudera, Inc. All rights reserved. 12 https://www.datainmotion.dev/2021/01/flank-real-time-transit-information-for.html
- 13. © 2021 Cloudera, Inc. All rights reserved. 13 Flink SQL Examples INSERT INTO weathernj SELECT CAST(`location` as STRING) `location`, CAST(station_id as STRING) station_id, latitude, longitude, CAST(observation_time as STRING) observation_time, CAST(weather as STRING) weather, CAST(temperature_string as STRING) temperature_string, temp_f, temp_c, relative_humidity, CAST(wind_string as STRING) wind_string, CAST(wind_dir as STRING) wind_dir, wind_degrees, wind_mph, wind_kt, pressure_in, CAST(dewpoint_string as STRING) dewpoint_string, dewpoint_f, dewpoint_c FROM weather WHERE `location` is not null and `location` <> 'null' and trim(`location`) <> '' and `location` like '%NJ';
- 14. © 2021 Cloudera, Inc. All rights reserved. 14 Flink SQL Examples INSERT INTO global_sensor_events SELECT scada.uuid, scada.systemtime , scada.temperaturef , scada.pressure , scada.humidity , scada.lux , scada.proximity , scada.oxidising , scada.reducing , scada.nh3 , scada.gasko,energy.`current`, energy.voltage ,energy.`power` , Energy.`total`,energy.fanstatus FROM energy, scada WHERE scada.systemtime >= energy.systemtime;
- 15. © 2021 Cloudera, Inc. All rights reserved. 15
- 16. © 2021 Cloudera, Inc. All rights reserved. 16 Apache NiFi DevOps https://nifi.apache.org/docs/nifi-docs/html/tool kit-guide.html#nifi_CLI
- 17. © 2021 Cloudera, Inc. All rights reserved. 17 Apache NiFi DevOps Make sure you check out Apache NiFi 1.13.2. There are some great new features. https://www.datainmotion.dev/2021/02/new-features -of-apache-nifi-1130.html Today we will discuss some DevOps around NiFi with NiFi CLI, REST and NiPyAPI. Some cool tools in the toolkit like diagnostics, threaddump. https://www.datainmotion.dev/2019/11/nifi-toolkit-cli- for-nifi-110.html
- 18. © 2021 Cloudera, Inc. All rights reserved. 18 Apache NiFi REST API https:/ /nifi.apache.org/docs/nifi-docs/rest-api/index.html {"clusterSummary":{"connectedNodes":"1 / 1","connectedNodeCount":1,"totalNodeCoun t":1,"connectedToCluster":true,"clustered":tr ue}}
- 19. © 2021 Cloudera, Inc. All rights reserved. 19 Apache NiFi REST API https:/ /nifi.apache.org/docs/nifi-docs/rest-api/index.html /nifi-api/flow/cluster/summary
- 20. © 2021 Cloudera, Inc. All rights reserved. 20 http://localhost:8080/nifi-api/resources http://localhost:8080/nifi-api/flow/cluster/summary http://localhost:8080/nifi-api/flow/config http://localhost:8080/nifi-api/flow/controller/bulletins http://localhost:8080/nifi-api/flow/history/?offset=1&count=100 http://localhost:8080/nifi-api/flow/processor-types http://localhost:8080/nifi-api/flow/reporting-tasks http://localhost:8080/nifi-api/flow/status nifi-api/system-diagnostics nifi-api/flow/controller/bulletins nifi-api/flow/process-groups/root nifi-api/flow/process-groups/root/controller-services nifi-api/flow/process-groups/root/status nifi-api/flow/process-groups/7a01d441-0164-1000-ec7a-54109819f084 Apache NiFi REST API - Examples
- 21. © 2021 Cloudera, Inc. All rights reserved. 21 Apache NiFi DevOps Some of this is a bit tricky. I will show you what we can do with DataFlow Experience in a future meetup. Time to hit the command line. cli.sh nifi pg-enable-services -u http:/ /edge2ai-1.dim.local:8080 --processGroupId root
- 22. © 2021 Cloudera, Inc. All rights reserved. 22 Apache NiFi CLI nifi pg-list -u http:/ /edge2ai-1.dim.local:8080
- 23. © 2021 Cloudera, Inc. All rights reserved. 23 CLI Tips You can run interactive or command at a time. For output: ● -ot simple ● -ot json You can loop calls with Shell Script, DevOps tools, NiFi and more.
- 24. © 2021 Cloudera, Inc. All rights reserved. 24 CLI Overall Commands nifi cluster-summary -u http://localhost:8080 nifi get-services -u http://localhost:8080 nifi get-reporting-task --baseUrl http://edge2ai-1.dim.local:8080 -verbose --reportingTaskId 07914d9f-1ce3-1174-0000-000039db6547 -ot json nifi get-reporting-tasks --baseUrl http://edge2ai-1.dim.local:8080 -verbose https://github.com/tspannhw/CloudDemo2021
- 25. © 2021 Cloudera, Inc. All rights reserved. 25 CLI Process Groups Commands nifi pg-list -u http://localhost:8080 nifi pg-enable-services -u http://edge2ai-1.dim.local:8080 --processGroupId root nifi pg-status -u http://localhost:8080 --processGroupId 6608ff51-89bb-3d66-4caf-f86ecaea950d nifi pg-status -u http://localhost:8080 --processGroupId root nifi pg-get-services -u http://localhost:8080 --processGroupId root nifi pg-enable-services -u http://localhost:8080 --processGroupId root nifi pg-start -u http://edge2ai-1.dim.local:8080 -pgid 2c1860b3-7f21-36f4-a0b8-b415c652fc62
- 26. © 2021 Cloudera, Inc. All rights reserved. 26 CLI Parameter Contexts Commands nifi list-param-contexts -u http://edge2ai-1.dim.local:8080 -verbose nifi pg-set-param-context -u http://edge2ai-1.dim.local:8080 -verbose -pgid 2c1860b3-7f21-36f4-a0b8-b415c652fc62 -pcid 39f0f296-0177-1000-ffff-ffffdccb6d90 nifi export-param-context -u http://localhost:8080 -verbose --paramContextId 8067d863-016e-1000-f0f7-265210d3e7dc nifi import-param-context -u http://localhost:8080 -i $f
- 27. © 2021 Cloudera, Inc. All rights reserved. 27 CLI Registry Commands registry list-buckets -u http://edge2ai-1.dim.local:18080 -verbose registry create-flow -verbose -u http://server:18080 -b 250a5ae5-ced8-4f4e-8b3b-01eb9d47a0d9 --flowName iotFlow registry import-flow-version -verbose -u http://server:18080 -f a5a4ac59-9aeb-416e-937f-e601ca8beba9 -i iot-1.json registry list-flows -u http://server:18080 -b 250a5ae5-ced8-4f4e-8b3b-01eb9d47a0d9 registry list-flows -bucketId 36cb79a4-f735-4f77-ba55-606718a9c3c9 -u http://localhost:18080 registry export-flow-version -f 5ebc2183-954e-4887-a28c-9d0ee54a02ed -o rainbow.json -ot json https://dzone.com/articles/devops-for-apache-nifi-17-and-more
- 28. © 2021 Cloudera, Inc. All rights reserved. 28 Apache NiFi DevOps ● https://github.com/tspannhw/CloudDemo2021 ● https://www.datainmotion.dev/2021/01/automating-starting-services-in-apache.html ● https://github.com/tspannhw/EverythingApacheNiFi ● https://github.com/tspannhw/BackupRegistry ● https://dev.to/tspannhw/backup-and-restore-nifi-registry-templates-14m ● https://dev.to/tspannhw/using-nifi-cli-to-restore-nifi-flows-from-backups-18p9 ● https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html#nifi_CLI ● https://nipyapi.readthedocs.io/en/latest/ ● https://www.datainmotion.dev/2019/04/simple-apache-nifi-operations-dashboard.html ● https://www.datainmotion.dev/2020/09/devops-working-with-parameter-contexts.html
- 29. © 2021 Cloudera, Inc. All rights reserved. 29 CLOUDERA DATAFLOW DATA-IN-MOTION PLATFORM
- 30. © 2021 Cloudera, Inc. All rights reserved. 30 CLOUDERA FLOW AND EDGE MANAGEMENT Enable easy ingestion, routing, management and delivery of any data anywhere (Edge, cloud, data center) to any downstream system with built in end-to-end security and provenance Advanced tooling to industrialize flow development (Flow Development Life Cycle) ACQUIRE • Over 300 Prebuilt Processors • Easy to build your own • Parse, Enrich & Apply Schema • Filter, Split, Merger & Route • Throttle & Backpressure FTP SFTP HL7 UDP XML HTTP EMAIL HTML IMAGE SYSLOG PROCESS HASH MERGE EXTRACT DUPLICATE SPLIT ENCRYPT TALL EVALUATE EXECUTE GEOENRICH SCAN REPLACE TRANSLATE CONVERT ROUTE TEXT ROUTE CONTENT ROUTE CONTEXT ROUTE RATE DISTRIBUTE LOAD DELIVER • Guaranteed Delivery • Full data provenance from acquisition to delivery • Diverse, Non-Traditional Sources • Eco-system integration FTP SFTP HL7 UDP XML HTTP EMAIL HTML IMAGE SYSLOG
- 31. © 2021 Cloudera, Inc. All rights reserved. 31
- 32. © 2021 Cloudera, Inc. All rights reserved. 32 TH N Y U