Apache Nifi Crash Course
1 like593 views
The document provides an overview of Apache NiFi and data flow management, explaining key concepts such as flow-based programming, data pipelines, and the challenges of moving large data sets efficiently. It highlights NiFi's features like guaranteed delivery, data provenance, and its integration capabilities with over 274 processors. Additionally, it discusses the latest updates to NiFi and introduces the NiFi registry for version control and asset management.
![© Hortonworks Inc. 2011–2018. All rights reserved9
• Standards
• Formats
• Protocols
• Veracity
• Validity
• Schemas
• Partitioning/Bun
dling
Data
Dataflow Challenges in 3 Categories
Infrastructure
• “Exactly Once”
Delivery
• Ensuring
Security
• Overcoming
Security
• Credential
Management
• Network
People
• Compliance
• “That
[person|team|g
roup]”
• Consumers
Change
• Requirements
Change
• “Exactly Once”
Delivery](https://image.slidesharecdn.com/1100ambrasbasahapachenificrashcourseandylopresto1-181017202403/85/Apache-Nifi-Crash-Course-9-320.jpg)
Apache Nifi Crash Course
- 1. © Hortonworks Inc. 2011–2018. All rights reserved1 Apache NiFi Crash Course Andy LoPresto | @yolopey Sr. Member of Technical Staff at Hortonworks, Apache NiFi PMC & Committer 11 October 2018 Dataworks Summit Singapore
- 2. © Hortonworks Inc. 2011–2018. All rights reserved2 Gauging Audience Familiarity with NiFi “What’s a NeeFee?” No experience with dataflow No experience with NiFi “I can pick this up pretty quickly” Some experience with dataflow Some experience with NiFi “I refactored the Ambari integration endpoint to allow for mutual authentication TLS during my coffee break” Forgotten more about NiFi than most of us will ever know
- 3. © Hortonworks Inc. 2011–2018. All rights reserved3 Agenda • Introduction • What is dataflow? • What is NiFi? • What’s next? • All slides provided online, so no need to transcribe
- 4. © Hortonworks Inc. 2011–2018. All rights reserved4 What Is Dataflow?
- 5. © Hortonworks Inc. 2011–2018. All rights reserved5 What Is Dataflow? • Moving some content from A to B • Content could be any bytes • Logs • HTTP • XML • CSV • Images • Video • Telemetry Producers A.K.A Things Anything AND Everything Internet! Consumers • User • Storage • System • …More Things
- 6. © Hortonworks Inc. 2011–2018. All rights reserved6 Connecting Data Points Is Easy • Simple enough to write a process • Bash/Ruby/Python • SQL proc • etc. Log files SQL Big Data
- 7. © Hortonworks Inc. 2011–2018. All rights reserved7 Big Data Is About Scale… • …and this doesn’t scale • Example use case: • AOL Data Processing • AWS -> HDFS • 20 TB ingested/day • Lev Brailovskiy, “Data Ingestion and Distribution with Apache NiFi”, Slide 27, 02/2017 • https://www.slideshare.net/LevBr ailovskiy/data-ingestion-and- distribution-with-apache-nifi
- 8. © Hortonworks Inc. 2011–2018. All rights reserved8 Moving Data Effectively Is Hard “Data Pipeline” https://xkcd.com/2054/
- 9. © Hortonworks Inc. 2011–2018. All rights reserved9 • Standards • Formats • Protocols • Veracity • Validity • Schemas • Partitioning/Bun dling Data Dataflow Challenges in 3 Categories Infrastructure • “Exactly Once” Delivery • Ensuring Security • Overcoming Security • Credential Management • Network People • Compliance • “That [person|team|g roup]” • Consumers Change • Requirements Change • “Exactly Once” Delivery
- 10. © Hortonworks Inc. 2011–2018. All rights reserved10 Raise your hand if you want to maintain Python scripts for the rest of your life Let’s Connect Lots of As to Bs to As to Cs to Bs to Δs to Cs to ϕs
- 11. © Hortonworks Inc. 2011–2018. All rights reserved11 What Is Apache NiFi?
- 12. © Hortonworks Inc. 2011–2018. All rights reserved12 NiFi Is Based on Flow Based Programming (FBP) FBP Term NiFi Term Description Information Packet FlowFile Each object moving through the system. Black Box FlowFile Processor Performs the work, doing some combination of data routing, transformation, or mediation between systems. Bounded Buffer Connection The linkage between processors, acting as queues and allowing various processes to interact at differing rates. Scheduler Flow Controller Maintains the knowledge of how processes are connected, and manages the threads and allocations thereof which all processes use. Subnet Process Group A set of processes and their connections, which can receive and send data via ports. A process group allows creation of entirely new component simply by composition of its components.
- 13. © Hortonworks Inc. 2011–2018. All rights reserved13 • Guaranteed delivery • Data buffering • Backpressure • Pressure release • Prioritized queuing • Flow specific QoS • Latency vs. throughput • Loss tolerance Key Features Apache NiFi • Data provenance • Supports push and pull models • Recovery/recording a rolling log of fine-grained history • Visual command and control • Flow templates • Pluggable, multi-tenant security • Designed for extension • Clustering
- 14. © Hortonworks Inc. 2011–2018. All rights reserved14 Flowfiles Are Like HTTP Data HTTPData FlowFile HTTP/1.1 200 OK Date: Sun, 10 Oct 2010 23:26:07 GMT Server: Apache/2.2.8 (CentOS) OpenSSL/0.9.8g Last-Modified: Sun, 26 Sep 2010 22:04:35 GMT ETag: "45b6-834-49130cc1182c0" Accept-Ranges: bytes Content-Length: 13 Connection: close Content-Type: text/html Hello world! Standard FlowFile Attributes Key: 'entryDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016' Key: 'lineageStartDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016' Key: 'fileSize’ Value: '23609' FlowFile Attribute Map Content Key: 'filename’ Value: '15650246997242' Key: 'path’ Value: './’ Binary Content * Header Content
- 15. © Hortonworks Inc. 2011–2018. All rights reserved15 User Interface Less of this… … more of this
- 16. © Hortonworks Inc. 2011–2018. All rights reserved16 Deeper Ecosystem Integration: 274+ Processors, 57 Controller Services Hash Extract Merge Duplicate Scan GeoEnrich Replace ConvertSplit Translate Route Content Route Context Route Text Control Rate Distribute Load Generate Table Fetch Jolt Transform JSON Prioritized Delivery Encrypt Tail Evaluate Execute All Apache project logos are trademarks of the ASF and the respective projects. Fetch HTTP Syslog Email HTML Image HL7 FTP UDP XML SFTP AMQP WebSocket Parse Records Convert Records
- 17. © Hortonworks Inc. 2011–2018. All rights reserved17 Extension / Integration Points NiFi Term Description Flow File Processor Push/Pull behavior. Custom UI Reporting Task Used to push data from NiFi to some external service (metrics, provenance, etc.) Controller Service Used to enable reusable components / shared services throughout the flow REST API Allows clients to connect to pull information, change behavior, etc.
- 18. © Hortonworks Inc. 2011–2018. All rights reserved18 Architecture OS/Host JVM Flow Controller Web Server Processor 1 Extension N FlowFile Repository Content Repository Provenance Repository Local Storage Standalone Cluster
- 19. © Hortonworks Inc. 2011–2018. All rights reserved19 NiFi Architecture – Repositories – Pass by Reference FlowFile Content Provenance F1à C1 C1 P1à F1 Excerpt of demo flow… What’s happening inside the repositories… BEFORE AFTER F2à C1 C1 P3à F2 – Clone (F1) F1à C1 P2à F1 – Route P1à F1 – Create
- 20. © Hortonworks Inc. 2011–2018. All rights reserved20 NiFi Architecture – Repositories – Copy on Write FlowFile Content Provenance F1à C1 C1 P1à F1 - CREATE Excerpt of demo flow… What’s happening inside the repositories… BEFORE AFTER F1à C1 F1.1à C2 C2 (encrypted) C1 (plaintext) P2à F1.1 - MODIFY P1à F1 - CREATE
- 21. © Hortonworks Inc. 2011–2018. All rights reserved21 Data Provenance Constrained High-latency Localized context Hybrid – cloud/on-premises Low-latency Global context Origin – attribution Replay – recovery Evolution of topologies Long retention Types of Lineage • Event • Configuration
- 22. © Hortonworks Inc. 2011–2018. All rights reserved22 • Previously, data had to be divided into individual flowfiles to perform work • CSV output with 50k lines would need to be split, operated on, re- merged • 1 + 50k + 50k + 1 flowfiles = 100k flowfiles Record Parsing
- 23. © Hortonworks Inc. 2011–2018. All rights reserved23 • Now flowfile content can contain many “record” elements • Read and write with *Reader and *Writer Controller Services • Perform lookups, routing, conversion, SQL queries, validation, and more… • 1 + 1 flowfiles = 2 flowfiles Record Parsing
- 24. © Hortonworks Inc. 2011–2018. All rights reserved24 • Every provenance event record is encrypted with AES G/CM before being persisted to disk • Decrypted on deserialization for retrieval/query • Random access via offset seek • Handles key migration & rotation Encrypted Provenance Repository
- 25. © Hortonworks Inc. 2011–2018. All rights reserved25 What’s Next?
- 26. © Hortonworks Inc. 2011–2018. All rights reserved26 • NiFi 1.8.0 — … Oct 2018 (170+ Jiras) • Jetty, DB improvements • Auto load-balancing queues • TLS Toolkit w/ external CA • Record processor improvements • MiNiFi C++ 0.5.0 — 6 June 2018 • MiNiFi Java 0.5.0 — 7 July 2018 • NiFi Registry 0.3.0 — 25 Sept 2018 Introducing Apache NiFi Registry New Announcements
- 27. © Hortonworks Inc. 2011–2018. All rights reserved27 • Previously, flows were exported via XML templates • Didn’t contain sensitive values • Couldn’t be updated in-place • No tracking system • NiFi Registry brings asset management as first-class citizen to NiFi • Flows can be versioned • Flows can be promoted between environments Introducing Apache NiFi Registry 0.3.0 NiFi Registry for Dataflows
- 28. © Hortonworks Inc. 2011–2018. All rights reserved28 Community Health
- 29. © Hortonworks Inc. 2011–2018. All rights reserved29 Apache NiFi site https://nifi.apache.org Subproject MiNiFi site https://nifi.apache.org/minifi/ Subscribe to and collaborate at dev@nifi.apache.org users@nifi.apache.org Submit Ideas or Issues https://issues.apache.org/jira/browse/NIFI Follow us on Twitter @apachenifi Learn More and Join Us
- 30. © Hortonworks Inc. 2011–2018. All rights reserved30 More NiFi Today… Title Time Room The First Mile – Edge and IoT Data Collection with Apache NiFi and MiNiFi 1400 - 1440 Orchard Dataflow Management From Edge to Core with Apache NiFi 1450 - 1330 Orchard
- 31. © Hortonworks Inc. 2011–2018. All rights reserved31 https://hortonworks.com/tutorial/analyze- transit-patterns-with-apache-nifi/
- 32. © Hortonworks Inc. 2011–2018. All rights reserved32 Thank You alopresto@hortonworks.com | alopresto@apache.org | @yolopey github.com/alopresto/slides