AIDevWorldApacheNiFi101
0 likes222 views
The document provides an introduction to Apache NiFi, detailing its architecture, best practices, and development tips for data flow management. It emphasizes the importance of optimizing workflows, managing flow definitions, and using the correct configurations to enhance performance and maintainability. Additionally, it highlights upcoming features in NiFi 2.0, including Python integration and a rules engine for improved development assistance.
AIDevWorldApacheNiFi101
- 1. Apache NiFi 101: Introduction and Best Practices Timothy Spann Principal Developer Advocate
- 3. 3 Tim Spann Twitter: @PaasDev // Blog: datainmotion.dev Principal Developer Advocate. Princeton Future of Data Meetup. ex-Pivotal, ex-Hortonworks, ex-StreamNative, ex-PwC, ex-HPE https://medium.com/@tspann https://github.com/tspannhw
- 4. © 2021 Cloudera, Inc. All rights reserved. 4 Future of Data - New York + Princeton + Virtual @PaasDev https://www.meetup.com/futureofdata-princeton/ https://www.meetup.com/futureofdata-newyork/ From Big Data to AI to Streaming to Containers to Cloud to Analytics to Cloud Storage to Fast Data to Machine Learning to Microservices to ...
- 5. FLaNK Stack Weekly This week in Apache NiFi, Apache Flink, Apache Kafka, Apache Spark, Apache Iceberg, Python, Java, AI, ML, LLM and Open Source friends. https://bit.ly/32dAJft
- 6. © 2019 Cloudera, Inc. All rights reserved. 6
- 8. 8 ● NiFi Cluster Architecture ● Content Repository ● Provenance Repository ● FlowFile Repository ● FlowFile, Attributes, Process Groups, Connections, Flow Controllers ● Controller Services ● Common Attributes (uuid, filename, path, file size, ...) ● Expression Language ● Relationships ● Bulletins ● Input Port ● Output Port ● Empty Queues ● Setting Warning Levels ● Funnels ● RecordPath ● Using Record Processors (Readers/Writers) ● NiFi REST API ● Handling Errors ● Parameter Context / Parameters ● Summary / Cluster / Bulletins ● Reporting Tasks ● Back pressure ● Prioritized Queues ● Load Balancing Strategies ● Prioritization ● Using Search ● Using Documentation ● Site-to-Site Communication / Remote Process Groups ● Extensions ● Scheduling ● Tailing Files ● Reading sFTP/FTP Files ● Wait and Notify ● RetryFlowFile Pattern ● NiFi Calcite SQL ● Using Jolt ● Using JsonPath ● Making REST Calls ● Receiving REST Calls ● LookupRecord ● Working with Caches ● Restarting Flows ● Pass by Reference ● Using Regular Expressions ● Funnels Basic Understanding of Cloudera Flow Management - Apache NiFi
- 9. 9 Do Not: ● Do not Put 1,000 Flows on one workspace. ● If your flow has hundreds of steps, this is a Flow Smell. Investigate why. ● Do not Use ExecuteProcess, ExecuteScripts or a lot of Groovy scripts as a default, look for existing processors ● Do not Use Random Custom Processors you find that have no documentation or are unknown. ● Do not forget to upgrade, if you are running anything before Apache NiFi 1.10, upgrade now! ● Do not run on default 512M RAM. ● Do not run one node and think you have a highly available cluster. ● Do not split a file with millions of records to individual records in one shot without checking available space/memory and back pressure. ● Use Split processors only as an absolute last resort. Many processors are designed to work on FlowFiles that contain many records or many lines of text. Keeping the FlowFiles together instead of splitting them apart can often yield performance that is improved by 1-2 orders of magnitude.
- 10. 10 Do: ● Reduce, Reuse, Recycle. Use Parameters to reuse common modules. ● Put flows, reusable chunks (write to Slack, Database, Kafka) into separate Process Groups. ● Write custom processors if you need new or specialized features ● Use RecordProcessors everywhere ● Read the Docs! ● Use the NiFi Registry for version control. ● Use NiFi CLI and DevOps for Migrations. ● Run a CDP NiFi Datahub or CFM managed 3 or more node cluster. ● Walk through your flow and make sure you understand every step and it’s easy to read and follow. Is every processor used? Are there dead ends? ● Do run Zookeeper on different nodes from Apache NiFi. ● For Cloud Hosted Apache NiFi - go with the "high cpu" instances, such as 8 cores, 7 GB ram. ● same flow 'templatized' and deployed many many times with different params in the same instance ● Use routing based on content and attributes to allow one flow to handle multiple nearly identical flows is better than deploying the same flow many times with tweaks to parameters in same cluster. ● Use the correct driver for your database. There's usually a couple different JDBC drivers.
- 11. © 2023 Cloudera, Inc. All rights reserved. 11 CLOUDERA FLOW MANAGEMENT - POWERED BY APACHE NiFi Ingest and manage data from edge-to-cloud using a no-code interface ● #1 data ingestion/movement engine ● Strong community ● Product maturity over 11 years ● Deploy on-premises or in the cloud ● Over 400+ pre-built processors ● Built-in data provenance ● Guaranteed delivery ● Throttling and Back pressure
- 12. CLOUD
- 13. © 2023 Cloudera, Inc. All rights reserved. 13 Development & Runtime of DataFlow Functions Step1. Develop functions on local workstation or in CDP Public Cloud using no-code, UI designer Step 2. Run functions on serverless compute services in AWS, Azure & GCP AWS Lambda Azure Functions Google Cloud Functions
- 14. 14 Flow Catalog • Central repository for flow definitions • Import existing NiFi flows • Manage flow definitions • Initiate flow deployments
- 15. 15 ReadyFlows • Cloudera provided flow definitions • Cover most common data flow use cases • Can be deployed and adjusted as needed • Made available through docs during Tech Preview
- 16. 16 Deployment Wizard • Turns flow definitions into flow deployments • Guides users through providing required configuration • Pick from pre-defined NiFi node sizes • Define KPIs for the deployment Start Deployment Wizard Provide Parameters Configure Sizing & Scaling Define KPIs
- 17. 17 Key Performance Indicators • Visibility into flow deployments • Track high level flow performance • Track in-depth NiFi component metrics • Defined in Deployment Wizard • Monitoring & Alerts in Deployment Details KPI Definition in Deployment Wizard KPI Monitoring
- 18. 18 Dashboard • Central Monitoring View • Monitors flow deployments across CDP environments • Monitors flow deployment health & performance • Drill into flow deployment to monitor system metrics and deployment events
- 19. 19 DATA FLOW DESIGN FOR EVERYONE • Cloud-native data flow development • Developers get their own sandbox • Start developing flows without installing NiFi • Redesigned visual canvas • Optimized interaction patterns • Integration into CDF-PC Catalog for versioning
- 20. NEW
- 22. New to 2023 Processors GenerateRecord GetAsanaObject PutSalesforceObject QuerySalesforceObject PutIoTDBRecord QueryIoTDBRecord https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020&version=12353320 ListGoogleDrive FetchGoogleDrive PutGoogleDrive PutBoxFile ListBoxFile FetchBoxFile PutDropbox DecryptContent DecryptContentCompatibility
- 23. New to 2023 Processors ExtractRecordSchema RemoveRecordField VerifyContentMAC TriggerHiveMetaStoreEvent “count” function added to RecordPath
- 24. AWS ML Service Processors https://github.com/tspannhw/FLaNK-AWSML
- 25. AWS Translate
- 26. 2.0!
- 28. 28
- 29. https://medium.com/cloudera-inc/getting-ready-for-apache-nifi-2-0-5a5e6a67f450 NiFi 2.0 Coming ● Python Integration ● Parameters ● JDK 17, maybe JDK 21+ ● JSON Flow Serialization ● Rules Engine for Development Assistance ● Run Process Group as Stateless ● flow.json.gz https://cwiki.apache.org/confluence/display/NIFI/NiFi+2.0+Release+Goals
- 30. Deprecating for Removal Deprecate Lua and Ruby Script Engines Deprecate ECMAScript Script Engine Deprecate the Ambari Reporting Task Deprecate Kafka 1.x components and 2.0 components XML Templates Variables See: https://cwiki.apache.org/confluence/display/NIFI/Deprecated+Components+and+Features
- 31. Start Using ExecuteStateless -> run your stateless flows right in a regular NiFi cluster Parameters JSON Flow Serialization Records everywhere
- 32. 32 Python as First Class (NIFI-11241) Graphical UI with custom Python based extensions NEW in NiFi 2.0
- 33. 33 Apache NiFi in a few numbers A very active project with a dynamic community & comparison with ACEU 2019 2800+ members on the Slack channel (535+ - 4 years ago) 475+ contributors on Github across the repositories (260+ - 4 years ago) 65 committers in the Apache NiFi community (45 - 4 years ago) Apache NiFi 1.23.2 is the latest release, NiFi 2.0 coming soon (NiFi 1.10 - 4 years ago) 14M+ docker pulls of the Apache NiFi image (1M+ - 4 years ago)
- 34. 34 © 2023 Cloudera, Inc. All rights reserved. Cloudera Edge Flow Manager (Command & Control of MiNiFi Agents) MiNiFi C++ (small footprint) MiNiFi Java (headless version of NiFi) NiFi Registry Cloudera NiFi for Kafka Connect NiFi in Cloudera DataFlow Functions Cloudera DataFlow Stateless NiFi NiFi Deploy Options from Open Source to Managed
- 35. 35 © 2023 Cloudera, Inc. All rights reserved. NiFi 2.0 is coming… https://medium.com/cloudera-inc/getting-ready-for-apache-nifi-2-0-5a5e6a67f450 - First-class citizen Python API - Rules Engine - NiFi Stateless at Process Group level - Java 21 (virtual threads, perf improvements, etc) https://medium.com/@george.vetticaden/accelerating-ai-data-pipelines-building-an-evernote-chatbot-with-apache-nifi-2-0-and-generative-ai-9d977466ff4c Closing the gap between data engineers and data scientists… - Export documentation (Sharepoint, OCR) to build the knowledge base powering your chatbot - Scrape the internet (Sitemap) to build the knowledge base powering your chatbot - Real-time streaming ingest of Slack to build the knowledge base powering your chatbot
- 36. WALKTHRU
- 37. 37 TH N Y U