Meet the Committers Webinar_ Lab Preparation
0 likes48 views
This document provides information and guidance for participants in a Cloudera DataFlow sandbox from May 3 to May 9, 2023. The key points are: - The sandbox is a shared environment where all users can access each other's flows. No production data should be used. Inactive flows may be stopped after 4 hours. - The sandbox will be destroyed on May 9 at midnight. Participants must complete and save their flows by then. - There is a best in flow competition to build and document a functional flow by the deadline for a chance to win a $2,000 prize. Requirements for eligible flows are described. - The document provides links to resources like ready flows, examples,
Meet the Committers Webinar_ Lab Preparation
- 1. Meet the Committers Lab Preparation 3-May-2023
- 2. Š 2023 Cloudera, Inc. All rights reserved. IMPORTANT NOTES Guidance Documentation Daily Zoom Sessions Examples Ready Flows Slack Channel Flow Proctors Shared Environment Since it is shared environment, each user has access to every other usersâ ďŹow. No production data should be used. We will stop your design sessions after 4 hours of inactivity.
- 3. Š 2023 Cloudera, Inc. All rights reserved. SANDBOX FROM MAY 3, 2023 to MAY 9 MIDNIGHT, 2023 Sandbox will be destroyed at midnight EST May 9, 2023 before May 10, 2023. You must complete your item, Save and Download Your Flows Before Then. All data and code will be destroyed on the end of the trial Submit your ďŹow (CRN), video and text via this form.
- 4. Š 2023 Cloudera, Inc. All rights reserved. NAVIGATE IN CHROME TO THE SHARED SANDBOX https://login.cdpworkshops.cloudera.com/auth/realms/se-workshop-5 /protocol/saml/clients/cdp-sso
- 5. Š 2023 Cloudera, Inc. All rights reserved. REGISTRATION Click Register Must use Recent Chrome Browser
- 10. Š 2023 Cloudera, Inc. All rights reserved. GETTING STARTED - GUIDED USE CASES â Syslog to Kafka topic â Reading and Filtering a Syslog Stream â Writing Critical Syslog Events to Apache Iceberg â Must use Recent Chrome Browser
- 11. Š 2023 Cloudera, Inc. All rights reserved. BEST IN FLOW COMPETITION - BUILD & DOCUMENT A FLOW A chance to win a $2,000 Amazon gift card. A great way to get recognition. Cloudera public award social media post.
- 12. Š 2023 Cloudera, Inc. All rights reserved. BEST IN FLOW COMPETITION - BUILD & DOCUMENT A FLOW A chance to win a $2,000 Amazon gift card. A great way to get recognition. Cloudera public award social media post.
- 13. Š 2023 Cloudera, Inc. All rights reserved. WHAT TO BUILD? You can extend or try one of our tutorials You can extend or use one of our Ready Flows You can connect to external resources (passwords are visible, only use public data or examples)
- 14. Š 2023 Cloudera, Inc. All rights reserved. FLOW REQUIREMENTS The following are the requirements for the ďŹow to be considered eligible for the competition: 1. The ďŹow must be developed using the new DataFlow Designer in the DataFlow Service sandbox. 2. The ďŹow must have at least one âsourceâ data. 3. The ďŹow must have at least one âdestinationâ where the data is delivered 4. The ďŹow must be functional, tested, and working using the Test session feature of the DataFlow Service. The Data viewer should be used to inspect the data payload within the different ďŹow steps. 5. The ďŹow must be checked into the DataFlow Catalog, deployed using the deployment wizard, and validated that it is correctly running. 6. Each submitted Flow must include the following additional details: â The CRN of the ďŹow was checked into the ďŹow catalog with a detailed description of the ďŹow and use case. â Link to a short blog describing the use case and the ďŹow that was built and deployed using DataFlow Designer â Link to a short video showing the ďŹow running in the Flow Designer with the test session and data traversing through ďŹow. The Data viewer should be used to inspect the data payload within the different ďŹow steps. â Product feedback on the DataFlow Service.
- 15. Š 2023 Cloudera, Inc. All rights reserved. Criteria Description Complete Flow Artifacts The submitted flow entry contains all the required artifacts, including Flow CRN in the Catalog, a link to the blog describing the use case and the flow, and a short video link showing the flow running with data traversing through the flow. Adheres to NiFi flow best practices Follows NiFi flow design best practices like record-oriented processors, controller services, and parameters. Showcases NiFi processing capabilities Showcases NiFi processing capabilities including protocol bridging, schema transformation, routing, filtering, enrichment, compression, etc. Universal Data Distribution The flow showcases multiple data sources and delivers data to multiple destinations. Uses the latest NiFi processors and controllers services Showcases the latest NiFi processors in the latest Apache NiFi release: 1.20, 1.19, 1.18, 1.17, including PutSnowflakeInternalStage, PutIceberg, UpdateDeltaLakeTable, Amazon ML Processors: Amazon Web Services Polly, Textract, Translate, and Transcribe services, etc. ReadyFlow The flow addresses a common data pipeline use case and can be reused by other users hence a good candidate to be added to the ReadyFlow gallery. Deployable The flow should be able to be deployed with minimum effort with the appropriate documentation (e.g.: description of parameters in the parameter context, the blog details, etc..)
- 16. Š 2023 Cloudera, Inc. All rights reserved. SANDBOX FLOW DEVELOPMENT BEST PRACTICES Uniquely Name your processors/ connections with yourid_ Parameterize connection information Donât use sensitive data in sandbox Donât use or change other peopleâs assets, only your own
- 17. Š 2023 Cloudera, Inc. All rights reserved. Donât use or change other peopleâs assets, only your own Reuse components via Copy and Process Groups We are here to help reach out via Slack or Zoom. SANDBOX FLOW DEVELOPMENT BEST PRACTICES
- 18. Š 2023 Cloudera, Inc. All rights reserved. DAILY ZOOM https://cloudera.zoom.us/j/964 60893376?pwd=eWZEVDhpZm pFSDNRejFzMXkvcHpOdz09
- 19. Š 2023 Cloudera, Inc. All rights reserved. SLACK CHANNEL https://bestinďŹow.slack.com/join /shared_invite/zt-1uj1ti8hc-8mnh mbr_AbOCD7f~A68P0w#/shared -invite/email
- 20. Š 2023 Cloudera, Inc. All rights reserved. SOURCE CODE AND EXAMPLES https://github.com/tspannhw/FLaN K-DataFlows
- 21. Š 2023 Cloudera, Inc. All rights reserved. Submit Your Flow https://docs.google.com/forms/d/1Ku2KSDFoxJy45jiOWuLRDi9Trpgm-42aaxeAVwy-fpo
- 22. Š 2023 Cloudera, Inc. All rights reserved. ADDITIONAL RESOURCES Cloudera DataFlow Designer: The Key to Agile Data Pipeline Development Streaming Data Ingestion into an Open Data Lakehouse Made Easy with DataFlow Example Cloudera DataFlow Designer: Kafka to Iceberg in Cloudera Data Warehouse Serverless NiFi Flows with DataFlow Functions DataFlow Functions Technical Demo DataFlow Documentation
- 23. 23 Š 2023 Cloudera, Inc. All rights reserved. Marketing Carolyn Duby Field CTO Field Meet the Data-In-Motion Team Tim Spann Developer Advocate Richard Walden DIM SME Lead Engineering Product Chris Joynt Product Marketing Joe Witt Engineering Leader George Vetticaden Product Leader Michael Kohs Product Owner for DataFlow Pierre Villard Product Owner for DataFlow Andre Araujo Product Owner for Stream Processing John Kuchmek DIM SME Expert
- 24. Š 2023 Cloudera, Inc. All rights reserved. WARNING âNotwithstanding any contrary terms in the Agreement, Customer acknowledges that information shared using the Trial Product is in a shared environment with similarly situated customers. All information in the shared environment is accessible by all other customers participating in the trial and such information will not be deemed ConďŹdential Information.â https://www.cloudera.com/legal/commercial-terms-and-conditio ns/cdp-public-cloud-trial-agreement.html
- 25. Š 2023 Cloudera, Inc. All rights reserved. CONTAINER BASED DATAFLOW Flow Deployment Flow Monitoring Allows easy ďŹow deployment based on NiFi 1.20 across CDP environments (Dev, QA, Prod) DeďŹne and assign KPIs to your ďŹows Easy NiFi version upgrades Update/Add KPIs, Update Parameters, Change sizing conďŹguration Automatic infrastructure scaling based on CPU utilization Central monitoring console for all your ďŹows across environments Monitor ďŹow metrics and infrastructure usage DeďŹne alerts for ďŹows breaching assigned KPIs Flow Catalog Keep track of your ďŹow deďŹnitions and versions in a central catalog Reuse your existing NiFi ďŹows by uploading them to the catalog Discover, search and reuse existing ďŹows easily
- 26. 26 Š 2023 Cloudera, Inc. All rights reserved. FLOW CATALOG ⢠Central repository for ďŹow deďŹnitions ⢠Import existing NiFi ďŹows ⢠Manage ďŹow deďŹnitions ⢠Initiate ďŹow deployments
- 27. 27 Š 2023 Cloudera, Inc. All rights reserved. TURNS FLOW DEFINITIONS INTO FLOW DEPLOYMENTS 2.) NiFi ConďŹg 4.) ConďŹgure Sizing & Scaling 5.) DeďŹne KPIs 1.) Start Deployment Wizard 3.) Provide Parameters for NiFi
- 28. 28 Š 2023 Cloudera, Inc. All rights reserved. KEY PERFORMANCE INDICATORS ⢠Visibility into ďŹow deployments ⢠Track high level ďŹow performance ⢠Track in-depth NiFi component metrics ⢠DeďŹned in Deployment Wizard ⢠Monitoring & Alerts in Deployment Details KPI DeďŹnition in Deployment Wizard KPI Monitoring
- 29. 29 Š 2023 Cloudera, Inc. All rights reserved. DASHBOARD ⢠Central Monitoring View ⢠Monitors ďŹow deployments across CDP environments ⢠Monitors ďŹow deployment health & performance ⢠Drill into ďŹow deployment to monitor system metrics and deployment events
- 30. 30 Š 2023 Cloudera, Inc. All rights reserved. DEPLOYMENT MANAGER ⢠Manage ďŹow deployment lifecycle (Suspend/Start/Terminate) ⢠Add/Edit KPIs ⢠Change sizing conďŹguration ⢠Update parameters ⢠Change NiFi version of the deployment ⢠Gateway to NiFi canvas
- 32. 32 Š 2023 Cloudera, Inc. All rights reserved. TH N Y U