You Can't Search Without Data
1 like676 views
The document presents an overview of Apache NiFi, a tool designed to facilitate global enterprise data flow by addressing challenges related to data formats, protocols, and security. It discusses key features of NiFi, including visual command and control, data lineage, prioritization, and integration with Solr for data management. Recent developments and future improvements in NiFi, such as enhanced record processing and a variable registry for better parameter handling, are also highlighted.
You Can't Search Without Data
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda à The Problem à Apache NiFi Overview à Integration between NiFi & Solr à Recent & Future Work à Demo Cool Stuff! à Q&A
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved About Me à Staff Software Engineer @ Hortonworks à Apache NiFi PMC & Committer à Contributed Solr processors in March 2015 – https://issues.apache.org/jira/browse/NIFI-461 à bbende@hortonworks.com / Twitter @bbende / bryanbende.com
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache NiFi • Created to address the challenges of global enterprise dataflow • Key features: – Visual Command and Control – Data Lineage (Provenance) – Data Prioritization – Data Buffering/Back-Pressure – Control Latency vs. Throughput – Secure Control Plane / Data Plane – Scale Out Clustering – Extensibility
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved NiFi Core Concepts FBP Term NiFi Term Description Information Packet FlowFile Each object moving through the system. Black Box FlowFile Processor Performs the work, doing some combination of data routing, transformation, or mediation between systems. Bounded Buffer Connection The linkage between processors, acting as queues and allowing various processes to interact at differing rates. Scheduler Flow Controller Maintains the knowledge of how processes are connected, and manages the threads and allocations thereof which all processes use. Subnet Process Group A set of processes and their connections, which can receive and send data via ports. A process group allows creation of entirely new component simply by composition of its components.
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Visual Command & Control • Drag & drop processors to build a flow • Start, stop, & configure components in real-time • View errors & corresponding messages • View statistics & health of the dataflow • Create shareable templates of common flows
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Provenance/Lineage • Tracks data at each point as it flows through the system • Records, indexes, and makes events available for display • Handles fan-in/fan-out, i.e. merging and splitting data • View attributes and content at given points in time
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Prioritization • Configure a prioritizer per connection • Determine what is important for your data – time based, arrival order, importance of a data set • Funnel many connections down to a single connection to prioritize across data sets
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Back-Pressure • Configure back-pressure per connection • Based on number of FlowFiles or total size of FlowFiles • Upstream processor no longer scheduled to run until below threshold
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Latency vs. Throughput • Choose between lower latency, or higher throughput on each processor • Higher throughput allows framework to batch together all operations for the selected amount of time for improved performance • Processor developer determines whether to support this by using @SupportsBatching annotation
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Architecture - Standalone OS/Host JVM Flow Controller Web Server Processor 1 Extension N FlowFile Repository Content Repository Provenance Repository Local Storage à FlowFile Repository – Write Ahead Log – State of every FlowFile – Pointers to content repository (pass-by-reference) à Content Repository – FlowFile content – Copy-on-write à Provenance Repository – Write Ahead Log + Lucene Indexes – Store & search lineage events
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved OS/Host JVM Flow Controller Web Server Processor 1 Extension N FlowFile Repository Content Repository Provenance Repository Local Storage OS/Host JVM Flow Controller Web Server Processor 1 Extension N FlowFile Repository Content Repository Provenance Repository Local Storage Architecture - Cluster OS/Host JVM Flow Controller Web Server Processor 1 Extension N FlowFile Repository Content Repository Provenance Repository Local Storage ZooKeeper à Same dataflow on each node, data partitioned across cluster à Access the UI from any node à ZooKeeper for auto-election of Cluster Coordinator & Primary Node à Cluster Coordinator receives heartbeats from other nodes, manages joining/ disconnecting à Primary Node for scheduling processors on a single node
- 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved NiFi Solr Processors à Support Solr Cloud and stand-alone Solr instances à Leverage SolrJ (CloudSolrClient & HttpSolrClient) à GetSolr – Extract new documents à PutSolrContentStream – Stream flow file content to an update handler
- 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved PutSolrContentStream à Choose Solr Type – Cloud or Standard à Specify ZooKeeper hosts, or the Solr URL with core à Specify the Solr path for the update handler à Dynamic Properties sent as key/value pairs on request à Relationships for success, failure, and connection failure
- 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved GetSolr à Incrementally extract new documents à Main query is *:*, Solr Query is optional filter query à Date Field used as filter query, from last execution or initial value à Sorted by date field and unique key à Cursor mark used behind the scenes à Specify return fields, or all if blank à Output Solr XML, or Records
- 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Interacting With a Secure Solr à Basic Auth – Provider username/password à Kerberos – Set JAAS system property in bootstrap.conf – Provide name of JAAS entry for processor to use à TLS/SSL – Provide an SSL Context Service – One-way TLS with Truststore only – Two-way TLS with Keystore + Truststore
- 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Problem – Conversion Between Data Formats à Specialized processors to operate on different data types à Sometimes missing conversions à Sometimes missing a specific function for a data type à Sometimes implemented with different libraries causing inconsistencies
- 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Solution – Record Processing à Introduce the concept of a ”record” – Released in Apache NiFi 1.2.0 (May 2017), improvements in 1.3.0 and 1.4.0 à Centralize the logic for reading/writing records into controller services – Readers/Writers for CSV, Json, Avro, etc. à Provide standard processors that operate on records – ConvertRecord, QueryRecord, PartitionRecord, UpdateRecord, etc. à Provide integration with schema registries – Local Schema Registry, Hortonworks Schema Registry, Confluent Schema Registry à Can still handle arbitrary data, but process records when appropriate
- 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Problem – Variable Handling à Need to parametrize values in the flow per environment – Connection strings, URLs, File System paths, etc. à Can set variables in bootstrap.conf – -Dmy.var=foo à Can set a properties file in nifi.properties – nifi.variable.registry.properties=production.properties à Both require command line access à Both require restart to pick up changes
- 29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Solution – First Class Variable Registry à Variables associated with a process group, released in 1.4.0 à Right-click on canvas to view variables for current group à Hierarchical order of precedence, resolve closest reference to component à Editing variables automatically restarts any components referencing the variables
- 30. 30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Problem – How do I deploy my flow? à Most organizations want the classic development lifecycle (dev -> int -> prod) à Can copy flow.xml.gz between environments – Requires copying entire data flow – Can’t tell what changed, hard to diff if you put in version control – Requires all environments use the same encryption key for sensitive properties à Can make templates for portions of the flow – Script creation of template and deployment to next environment – Requires stopping flow and removing components, then re-instantiating template – No easy way to see changes, hard to rollback
- 31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Solution – NiFi Registry à DISCLAIMER - UNDER DEVELOPMENT & NOT RELEASED YET! à Complimentary application, sub-project of Apache NiFi – https://github.com/apache/nifi-registry – https://issues.apache.org/jira/projects/NIFIREG à Central location for storage/management of shared resources across NiFi instances à Initial capability to store and retrieve “versioned flows” à A versioned flow is a snapshot of a process group at a given point in time à Potentially store extensions, shared data sets, and more in the future
- 33. 33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Example Scenario à User data – https://randomuser.me à Initially in CSV format – name.title,name.first,name.last,email,registered – mr,dennis,reyes,dennis.reyes@example.com,2012-04-10 01:54:19 – miss,carole,gomez,carole.gomez@example.com,2002-12-17 22:15:49 à Requirements – Convert CSV to JSON – Add a full_name field with first name + last name – Add a gender field based on title (i.e. if title == mr then MALE) – Ingest to different Solr collections depending on environment
- 35. 35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Learn more and join us! Apache NiFi site http://nifi.apache.org Subscribe to and collaborate at dev@nifi.apache.org users@nifi.apache.org Submit Ideas or Issues https://issues.apache.org/jira/browse/NIFI Follow us on Twitter @apachenifi