SlideShare a Scribd company logo

Turning a Data Pond into a Data Lake with Apache NiFi

Download as PPTX, PDF
G
Gene Peters

The document discusses Apache NiFi and how it can be used to turn data ponds into data lakes. It provides an overview of Apache NiFi, including that it was developed by the NSA to enable real-time data ingestion and routing. It also discusses NiFi's terminology, security features, and how it can be used for use cases like on-premise to cloud replication, data ingestion from REST APIs, and data conversion and transformation. The presentation concludes with a demo of starting a Google Cloud data lake using NiFi and a discussion of alternative tools.

Software
1 of 19
Downloaded 13 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
pagepageYour Company Name Presentation 1 Gene Peters Cask HQ 5.10.2017 Turning a Data Pond into a Data Lake With Apache NiFi
page Introduction Gene F. Peters Founder & CTO 2 Senior Data Engineer, Kixeye Consulting Engineer, Pebble, Tilt College of William & Mary
pagepage 3 The Evolution of a Data Pipeline: Starting Out
pagepage 4 The Evolution of a Data Pipeline: Growing More Complex
pagepage 5 The Evolution of a Data Pipeline: Enterprise Scale
pagepage 6 Enter Apache NiFi • Function: real-time data ingest and routing • Developed and open sourced by the NSA • v1.0 released August 30, 2016, v1.2.0 released May 8, 2017 • Hortonworks major sponsor behind this project
pagepage 7 Technology • Write-ahead transaction log for changes to content • At least once message guarantee • Horizontally Scalable • Single node or cluster deployment
pagepage 8 Terminology and API • Processors • Relationships • Controller Services • Process Groups
pagepage 9 Security • Control Plane • Pluggable Kerberos / LDAP integration • Per-component access control • Full usage auditing • Data Plane • SSL encryption of IPC • Encryption of data at rest
pagepage 10 Use Case: On-Premise -> Cloud Replication
pagepage 11 Use Case: Cloud to Cloud Replication
pagepage 12 Use Case: Data Ingest from REST APIs
pagepage 13 Use Case: Data Conversion and Transformation
pagepage 14 Use Case: RDB to NoSQL
pagepage 15 Putting it all together: A Data Lake in the Cloud
pagepage 16 Demo • Demo: Starting up a Google Cloud data lake with Salesforce and Postgres sources
pagepage 17 Alternatives • Streamsets • Airflow • Pentaho • Storm / Flink / Spark Streaming • Cloud-based migration tools • Homebuilt Code
page Questions and Discussion 18
page Contact Info facebook.com/telligentdata twitter.com/telligentdata Thank You info@telligent-data.com angel.co/telligent-data (415) 758-0155 19

More Related Content

PDF
Database, data storage, hosting with Firebase
Tu Pham
 
PPTX
NGINX MRA Fabric Model Release and Ask Me Anything Part 4
NGINX, Inc.
 
PPTX
Firebase - cloud based real time database
Glenn Bech
 
PPTX
Running OpenStack in Production
Tesora
 
PDF
Elastic Cloud Enterprise in Azure with Devon
Elasticsearch
 
PPTX
Leveraging OpenStack at Scale: How the Elastic Cloud Drives Innovation Velocity
Tesora
 
PPTX
Fast SAP system provisioning based on CloudStack
ShapeBlue
 
PDF
Unlocking the Cloud operating model with GitHub Actions
Mitchell Pronschinske
 
Database, data storage, hosting with Firebase
Tu Pham
 
NGINX MRA Fabric Model Release and Ask Me Anything Part 4
NGINX, Inc.
 
Firebase - cloud based real time database
Glenn Bech
 
Running OpenStack in Production
Tesora
 
Elastic Cloud Enterprise in Azure with Devon
Elasticsearch
 
Leveraging OpenStack at Scale: How the Elastic Cloud Drives Innovation Velocity
Tesora
 
Fast SAP system provisioning based on CloudStack
ShapeBlue
 
Unlocking the Cloud operating model with GitHub Actions
Mitchell Pronschinske
 

What's hot (20)

PPTX
CloudStack EU user group - fast SAP provisioning
ShapeBlue
 
PPTX
Succeeding with OpenStack in the Enterprise (OpenStack Summit Austin 2016)
Omri Gazitt
 
PPTX
OpenStack at Bloomberg
Tesora
 
PPTX
Introduction to Firebase
Mustafa Şenel
 
PDF
Elastic on a Hyper-Converged Infrastructure for Operational Log Analytics
Elasticsearch
 
PPTX
What's Next for OpenStack at Walmart
Tesora
 
PDF
Consul by Mitchell - HashiCorp Meetup
hashicorpjp
 
PDF
MongoDB World 2016: Get MEAN and Lean with MongoDB and Kubernetes
MongoDB
 
PDF
Better Search and Business Analytics at Southern Glazer’s Wine & Spirits
Elasticsearch
 
PDF
Building the future of Digital Television and Enterprise Database Management ...
MongoDB
 
PDF
Managing Content with the Nuxeo Platform - CXP ECM Event
Nuxeo
 
PDF
R, Spark, Tensorflow, H20.ai Applied to Streaming Analytics
Kai Wähner
 
PDF
Monitoring IPv6 Networks
ThousandEyes
 
PPTX
Trilio for OpenStack; Protect, Recover, Migrate
Tom Lahive
 
PDF
Building a reliable and cost effect logging system at Box
Elasticsearch
 
PDF
CSX: Real-time Business Discovery with the Elastic Stack
Elasticsearch
 
PDF
Lenovo: Elastic Stack Practices in Enterprise Integration
Elasticsearch
 
PPTX
2013 Collaboration Tour - Keynote
Perforce
 
PPTX
Cloud stack user group - Welcome
ShapeBlue
 
PDF
Search for All with Elastic Workplace Search
Elasticsearch
 
CloudStack EU user group - fast SAP provisioning
ShapeBlue
 
Succeeding with OpenStack in the Enterprise (OpenStack Summit Austin 2016)
Omri Gazitt
 
OpenStack at Bloomberg
Tesora
 
Introduction to Firebase
Mustafa Şenel
 
Elastic on a Hyper-Converged Infrastructure for Operational Log Analytics
Elasticsearch
 
What's Next for OpenStack at Walmart
Tesora
 
Consul by Mitchell - HashiCorp Meetup
hashicorpjp
 
MongoDB World 2016: Get MEAN and Lean with MongoDB and Kubernetes
MongoDB
 
Better Search and Business Analytics at Southern Glazer’s Wine & Spirits
Elasticsearch
 
Building the future of Digital Television and Enterprise Database Management ...
MongoDB
 
Managing Content with the Nuxeo Platform - CXP ECM Event
Nuxeo
 
R, Spark, Tensorflow, H20.ai Applied to Streaming Analytics
Kai Wähner
 
Monitoring IPv6 Networks
ThousandEyes
 
Trilio for OpenStack; Protect, Recover, Migrate
Tom Lahive
 
Building a reliable and cost effect logging system at Box
Elasticsearch
 
CSX: Real-time Business Discovery with the Elastic Stack
Elasticsearch
 
Lenovo: Elastic Stack Practices in Enterprise Integration
Elasticsearch
 
2013 Collaboration Tour - Keynote
Perforce
 
Cloud stack user group - Welcome
ShapeBlue
 
Search for All with Elastic Workplace Search
Elasticsearch
 
Ad

Similar to Turning a Data Pond into a Data Lake with Apache NiFi (20)

PPTX
Open Marketing Meeting 03/27/2013
OpenStack
 
PDF
Moving to hyper-converged? Don’t forget about data protection.
Veritas Technologies LLC
 
PPTX
Integrating Apache Spark and NiFi for Data Lakes
DataWorks Summit/Hadoop Summit
 
PDF
Using Apache NiFi with Apache Pulsar for Fast Data On-Ramp
Timothy Spann
 
PDF
VMworld 2013: Beyond Mission Critical: Virtualizing Big-Data, Hadoop, HPC, Cl...
VMworld
 
PPTX
Lessons Learned Migrating from IBM BigInsights to Hortonworks Data Platform
DataWorks Summit
 
PDF
451 Research: Data Is the Key to Friction in DevOps
Delphix
 
PPTX
Unleash the Power of Equinix: Digital Transformation through Interconnection
Equinix
 
PPTX
OpenStack Grizzly Release
OpenStack
 
PDF
Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...
confluent
 
PPTX
Analytics at the Speed of Thought: Actian Express Overview
Actian Corporation
 
PPTX
Yaron Haviv, Iguaz.io - OpenStack and BigData - OpenStack Israel 2015
Cloud Native Day Tel Aviv
 
PDF
Considering bare metal as a viable cloud option
Internap
 
PPTX
GraphTour - Neo4j Database Overview
Neo4j
 
PPTX
Delivering High Performance Websites with NGINX
NGINX, Inc.
 
PDF
It's Prime Time for OpenStack--What Are You Waiting For?
Veritas Technologies LLC
 
PPTX
10 Reasons Snowflake Is Great for Analytics
Senturus
 
PDF
HBase Meetup @ Cask HQ 09/25
Cask Data
 
PPTX
Icehouse Release Webinar
OpenStack Foundation
 
PPTX
Integração de Dados com Apache NIFI - Marco Garcia Cetax
Marco Garcia
 
Open Marketing Meeting 03/27/2013
OpenStack
 
Moving to hyper-converged? Don’t forget about data protection.
Veritas Technologies LLC
 
Integrating Apache Spark and NiFi for Data Lakes
DataWorks Summit/Hadoop Summit
 
Using Apache NiFi with Apache Pulsar for Fast Data On-Ramp
Timothy Spann
 
VMworld 2013: Beyond Mission Critical: Virtualizing Big-Data, Hadoop, HPC, Cl...
VMworld
 
Lessons Learned Migrating from IBM BigInsights to Hortonworks Data Platform
DataWorks Summit
 
451 Research: Data Is the Key to Friction in DevOps
Delphix
 
Unleash the Power of Equinix: Digital Transformation through Interconnection
Equinix
 
OpenStack Grizzly Release
OpenStack
 
Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement ...
confluent
 
Analytics at the Speed of Thought: Actian Express Overview
Actian Corporation
 
Yaron Haviv, Iguaz.io - OpenStack and BigData - OpenStack Israel 2015
Cloud Native Day Tel Aviv
 
Considering bare metal as a viable cloud option
Internap
 
GraphTour - Neo4j Database Overview
Neo4j
 
Delivering High Performance Websites with NGINX
NGINX, Inc.
 
It's Prime Time for OpenStack--What Are You Waiting For?
Veritas Technologies LLC
 
10 Reasons Snowflake Is Great for Analytics
Senturus
 
HBase Meetup @ Cask HQ 09/25
Cask Data
 
Icehouse Release Webinar
OpenStack Foundation
 
Integração de Dados com Apache NIFI - Marco Garcia Cetax
Marco Garcia
 
Ad

Recently uploaded (20)

PPTX
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
PDF
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Andre Kraus
 
PDF
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
PDF
AI in Product Development-omnex systems
omnex systems
 
PDF
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
PDF
System and Network Administraation Chapter 3
jaramharo
 
PPT
Introduction Database Management System for Course Database
ReinertYosua
 
PDF
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
OnePlan Solutions
 
PPTX
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
IT IDOL Technologies
 
PPTX
Operating system designcfffgfgggggggvggggggggg
rmskumara09
 
PDF
Odoo Companies in India – Driving Business Transformation.pdf
azhuhardeen1968
 
PDF
2025 Textile ERP Trends: SAP, Odoo & Oracle
Aetos Technologies
 
PPTX
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
PDF
System and Network Administration Chapter 2
jaramharo
 
PDF
Digital Strategies for Manufacturing Companies
Virendra Rai, PMP
 
PDF
Audit Checklist Design Aligning with ISO, IATF, and Industry Standards — Omne...
rohnjim07
 
PPTX
Odoo POS Development Services by CandidRoot Solutions
CandidRoot Solutions Private Limited
 
PDF
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
PDF
Upgrade and Innovation Strategies for SAP ERP Customers
Virendra Rai, PMP
 
PPTX
Online Work Permit System for Fast Permit Processing
SHEQ Network Limited
 
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Andre Kraus
 
Adobe Illustrator 28.6 Crack My Vision of Vector Design
ghrom2211g
 
AI in Product Development-omnex systems
omnex systems
 
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
System and Network Administraation Chapter 3
jaramharo
 
Introduction Database Management System for Course Database
ReinertYosua
 
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
OnePlan Solutions
 
Agentic AI : A Practical Guide. Undersating, Implementing and Scaling Autono...
IT IDOL Technologies
 
Operating system designcfffgfgggggggvggggggggg
rmskumara09
 
Odoo Companies in India – Driving Business Transformation.pdf
azhuhardeen1968
 
2025 Textile ERP Trends: SAP, Odoo & Oracle
Aetos Technologies
 
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
System and Network Administration Chapter 2
jaramharo
 
Digital Strategies for Manufacturing Companies
Virendra Rai, PMP
 
Audit Checklist Design Aligning with ISO, IATF, and Industry Standards — Omne...
rohnjim07
 
Odoo POS Development Services by CandidRoot Solutions
CandidRoot Solutions Private Limited
 
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
Upgrade and Innovation Strategies for SAP ERP Customers
Virendra Rai, PMP
 
Online Work Permit System for Fast Permit Processing
SHEQ Network Limited
 

Turning a Data Pond into a Data Lake with Apache NiFi