SlideShare a Scribd company logo

Big data and hadoop training - Session 5

Download as PPTX, PDF
H
hkbhadraa

This document discusses setting up a Kafka cluster on Amazon Web Services for a big data pipeline. It provides instructions for launching a 3 node Kafka cluster with Zookeeper and two Kafka brokers. Configuration details are given for each broker node and steps are outlined for starting the Zookeeper and Kafka processes, creating a topic, and running a consumer to read from the topic. The purpose of the Kafka cluster is to work with streaming data in a Python Spark application as part of the big data pipeline.

Data & Analytics
1 of 14
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Big Data and Hadoop Training Session 5
Big Data - Pipeline Big Data Pipeline Lambda Architecture - Streaming(Real-Time) Layer with Apache Kafka Apache Hadoop Apache Spark Apache Cassandra on Amazon Web Services Cloud Platform
Big Data - Pipeline
Big Data - Pipeline 3 EC2 instance for Kafka Cluster
Big Data - Pipeline Repeat commands for all - 3 EC2 instance for Kafka Cluster cat /etc/*-release sudo add-apt-repository ppa:webupd8team/java sudo apt-get update sudo apt-get install oracle-java8-installer java -version mkdir kafka cd kafka wget http://download.nextag.com/apache/kafka/0.10.0.0/kafka_2.11-0.10.0.0.tgz tar -zxvf kafka_2.11-0.10.0.0.tgz cd kafka_2.11-0.10.0.0 ZooKeeper ==> 172.31.48.208 / 52.91.1.93 Kafka-datanode1 ==> 172.31.63.203 / 54.173.215.211 Kafka-datanode2 ==> 172.31.9.25 / 54.226.29.194
Big Data - Pipeline Kafka-datanode1 (set following properties for config/server.properties) ubuntu@ip-172-31-63-203:~/kafka/kafka_2.11-0.10.0.0$ vi config/server.properties broker.id=1 listeners=PLAINTEXT://172.31.63.203:9092 advertised.listeners=PLAINTEXT://54.173.215.211:9092 zookeeper.connect=52.91.1.93:2181 Kafka-datanode2 (set following properties for config/server.properties) ubuntu@ip-172-31-9-25:~/kafka/kafka_2.11-0.10.0.0$ vi config/server.properties broker.id=2 listeners=PLAINTEXT://172.31.9.25:9092 advertised.listeners=PLAINTEXT://54.226.29.194:9092 zookeeper.connect=52.91.1.93:2181 Modify config/server.properties for kafka-datanode1 & kafkadatanode2 ZooKeeper ==> 172.31.48.208 / 52.91.1.93 Kafka-datanode1 ==> 172.31.63.203 / 54.173.215.211 Kafka-datanode2 ==> 172.31.9.25 / 54.226.29.194
Big Data - Pipeline Launch zookeeper / datanode1 / datanode2 ZooKeeper ==> 172.31.48.208 / 52.91.1.93 Kafka-datanode1 ==> 172.31.63.203 / 54.173.215.211 Kafka-datanode2 ==> 172.31.9.25 / 54.226.29.194 1) Start zookeeper bin/zookeeper-server-start.sh config/zookeeper.properties 2) Start server on Kafka-datanode1 bin/kafka-server-start.sh config/server.properties 3) Start server on Kafka-datanode2 bin/kafka-server-start.sh config/server.properties export KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" 4) Create Topic & Start consumer bin/kafka-topics.sh --zookeeper 52.91.1.93:2181 --create --topic data --partitions 1 --replication-factor 2 bin/kafka-console-consumer.sh --zookeeper 52.91.1.93:2181 --topic data --from-beginning
Big Data - Pipeline Launch Kafka Cluster (Zookeeper/kafka datanode1/ kafka datanode2)
Big Data - Pipeline Execute Python / Kafka Spark Job
Big Data - Pipeline Sample data which we will be sending to Kafka Server from Java Kafka Producer (csv file)
Big Data - Pipeline Python Spark Job Processing Data from AWS Kafka Cluster
Big Data - Pipeline Python Spark Streaming Application
Thank You
hkbhadraa@gmail.com

More Related Content

PDF
Setup 3 Node Kafka Cluster on AWS - Hands On
hkbhadraa
 
PDF
Spotify: Automating Cassandra repairs
DataStax Academy
 
PPTX
Big data lambda architecture - Streaming Layer Hands On
hkbhadraa
 
PDF
Set up Hadoop Cluster on Amazon EC2
IMC Institute
 
PDF
Open erp on ubuntu
Iker Coranti
 
PDF
Describing Kafka security in AsyncAPI
Dale Lane
 
PDF
Useful Kafka tools
Dale Lane
 
PPTX
Capistrano 3 Deployment
Creston Jamison
 
Setup 3 Node Kafka Cluster on AWS - Hands On
hkbhadraa
 
Spotify: Automating Cassandra repairs
DataStax Academy
 
Big data lambda architecture - Streaming Layer Hands On
hkbhadraa
 
Set up Hadoop Cluster on Amazon EC2
IMC Institute
 
Open erp on ubuntu
Iker Coranti
 
Describing Kafka security in AsyncAPI
Dale Lane
 
Useful Kafka tools
Dale Lane
 
Capistrano 3 Deployment
Creston Jamison
 

What's hot (16)

PDF
Cyber Range - An Open-Source Offensive / Defensive Learning Environment on AWS
Tom Cappetta
 
PPTX
Cyber Range - Blackhat Europe 19 Arsenal
Tom Cappetta
 
PPTX
Salting new ground one man ops from scratch
Jay Harrison
 
PDF
[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus
OpenStack Korea Community
 
PPTX
Enable IPv6 on Route53 AWS ELB, docker and node App
Fyllo
 
PDF
[2C4]Clustered computing with CoreOS, fleet and etcd
NAVER D2
 
PDF
Docker at OpenDNS
OpenDNS
 
PDF
SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...
SaltStack
 
PDF
Salt conf 2014-installing-openstack-using-saltstack-v02
Yazz Atlas
 
PPT
Python Deployment with Fabric
andymccurdy
 
PPTX
OpenShift4 Installation by UPI on kvm
Jooho Lee
 
PDF
Small, Simple, and Secure: Alpine Linux under the Microscope
Docker, Inc.
 
PDF
Multinode kubernetes-cluster
Ram Nath
 
PDF
Etcd- Mission Critical Key-Value Store
CoreOS
 
PDF
[오픈소스컨설팅] EFK Stack 소개와 설치 방법
Open Source Consulting
 
PPTX
What makes AWS invincible? from JAWS Days 2014
Emma Haruka Iwao
 
Cyber Range - An Open-Source Offensive / Defensive Learning Environment on AWS
Tom Cappetta
 
Cyber Range - Blackhat Europe 19 Arsenal
Tom Cappetta
 
Salting new ground one man ops from scratch
Jay Harrison
 
[OpenInfra Days Korea 2018] Day 2 - E6 - OpenInfra monitoring with Prometheus
OpenStack Korea Community
 
Enable IPv6 on Route53 AWS ELB, docker and node App
Fyllo
 
[2C4]Clustered computing with CoreOS, fleet and etcd
NAVER D2
 
Docker at OpenDNS
OpenDNS
 
SaltConf14 - Eric johnson, Google - Orchestrating Google Compute Engine with ...
SaltStack
 
Salt conf 2014-installing-openstack-using-saltstack-v02
Yazz Atlas
 
Python Deployment with Fabric
andymccurdy
 
OpenShift4 Installation by UPI on kvm
Jooho Lee
 
Small, Simple, and Secure: Alpine Linux under the Microscope
Docker, Inc.
 
Multinode kubernetes-cluster
Ram Nath
 
Etcd- Mission Critical Key-Value Store
CoreOS
 
[오픈소스컨설팅] EFK Stack 소개와 설치 방법
Open Source Consulting
 
What makes AWS invincible? from JAWS Days 2014
Emma Haruka Iwao
 
Ad

Similar to Big data and hadoop training - Session 5 (20)

DOCX
Kafk a with zoo keeper setup documentation
Thiyagarajan saminadane
 
PPTX
Real-time streaming and data pipelines with Apache Kafka
Joe Stein
 
PDF
Sparkstreaming
Marilyn Waldman
 
PDF
Continuous Delivery: The Next Frontier
Carlos Sanchez
 
PPTX
Simplifying Migration from Kafka to Pulsar - Pulsar Summit NA 2021
StreamNative
 
PDF
Drupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google Cloud
Dropsolid
 
PPTX
Sparkstreaming with kafka and h base at scale (1)
Sigmoid
 
PPTX
Journey to Microservice architecture via Amazon Lambda
Axilis
 
DOCX
Apache kafka configuration-guide
Chetan Khatri
 
PPTX
Couch to OpenStack: Nova - July, 30, 2013
Trevor Roberts Jr.
 
PDF
Real-time Data Pipeline: Kafka Streams / Kafka Connect versus Spark Streaming
Abdelhamide EL ARIB
 
PPTX
Kafka Rest.pptx
wonyong hwang
 
PPT
Spark Streaming Info
Doug Chang
 
PPTX
Training
HemantDunga1
 
PDF
Camel Kafka Connectors: Tune Kafka to “Speak” with (Almost) Everything (Andre...
HostedbyConfluent
 
PDF
Apache Kafka - Scalable Message-Processing and more !
Guido Schmutz
 
PDF
Running Kafka as a Native Binary Using GraalVM with Ozan Günalp
HostedbyConfluent
 
PDF
JavaCro'15 - Conquer the Internet of Things with Java and Docker - Johan Jans...
HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
 
PDF
Stream Processing using Apache Spark and Apache Kafka
Abhinav Singh
 
PDF
Apache Kafka - A modern Stream Processing Platform
Guido Schmutz
 
Kafk a with zoo keeper setup documentation
Thiyagarajan saminadane
 
Real-time streaming and data pipelines with Apache Kafka
Joe Stein
 
Sparkstreaming
Marilyn Waldman
 
Continuous Delivery: The Next Frontier
Carlos Sanchez
 
Simplifying Migration from Kafka to Pulsar - Pulsar Summit NA 2021
StreamNative
 
Drupaljam 2017 - Deploying Drupal 8 onto Hosted Kubernetes in Google Cloud
Dropsolid
 
Sparkstreaming with kafka and h base at scale (1)
Sigmoid
 
Journey to Microservice architecture via Amazon Lambda
Axilis
 
Apache kafka configuration-guide
Chetan Khatri
 
Couch to OpenStack: Nova - July, 30, 2013
Trevor Roberts Jr.
 
Real-time Data Pipeline: Kafka Streams / Kafka Connect versus Spark Streaming
Abdelhamide EL ARIB
 
Kafka Rest.pptx
wonyong hwang
 
Spark Streaming Info
Doug Chang
 
Training
HemantDunga1
 
Camel Kafka Connectors: Tune Kafka to “Speak” with (Almost) Everything (Andre...
HostedbyConfluent
 
Apache Kafka - Scalable Message-Processing and more !
Guido Schmutz
 
Running Kafka as a Native Binary Using GraalVM with Ozan Günalp
HostedbyConfluent
 
JavaCro'15 - Conquer the Internet of Things with Java and Docker - Johan Jans...
HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
 
Stream Processing using Apache Spark and Apache Kafka
Abhinav Singh
 
Apache Kafka - A modern Stream Processing Platform
Guido Schmutz
 
Ad

More from hkbhadraa (13)

PPTX
Big data and hadoop training - Session 3
hkbhadraa
 
PPTX
Big data and hadoop training - Session 2
hkbhadraa
 
PPTX
Retail products - machine learning recommendation engine
hkbhadraa
 
PPTX
Big data Lambda Architecture - Batch Layer Hands On
hkbhadraa
 
PDF
Project management part 5
hkbhadraa
 
PDF
Project management part 4
hkbhadraa
 
PDF
Project management part 3
hkbhadraa
 
PDF
Project management part 2
hkbhadraa
 
PDF
Project management part 1
hkbhadraa
 
PDF
Hadoop BIG Data - Fraud Detection with Real-Time Analytics
hkbhadraa
 
PDF
Gamification
hkbhadraa
 
PDF
Internet of things
hkbhadraa
 
PDF
IBM Bluemix Cloud Platform Application Development with Eclipse IDE
hkbhadraa
 
Big data and hadoop training - Session 3
hkbhadraa
 
Big data and hadoop training - Session 2
hkbhadraa
 
Retail products - machine learning recommendation engine
hkbhadraa
 
Big data Lambda Architecture - Batch Layer Hands On
hkbhadraa
 
Project management part 5
hkbhadraa
 
Project management part 4
hkbhadraa
 
Project management part 3
hkbhadraa
 
Project management part 2
hkbhadraa
 
Project management part 1
hkbhadraa
 
Hadoop BIG Data - Fraud Detection with Real-Time Analytics
hkbhadraa
 
Gamification
hkbhadraa
 
Internet of things
hkbhadraa
 
IBM Bluemix Cloud Platform Application Development with Eclipse IDE
hkbhadraa
 

Recently uploaded (20)

PPTX
IB Computer Science - Internal Assessment.pptx
agarwal18harsh08
 
PPTX
climate analysis of Dhaka ,Banglades.pptx
rrawjatun
 
PPTX
Computer network topology notes for revision
BatoolRawat
 
PPTX
STERILIZATION AND DISINFECTION-1.ppthhhbx
DivuuJain
 
PPTX
mbdjdhjjodule 5-1 rhfhhfjtjjhafbrhfnfbbfnb
CHINTU5000
 
PPTX
Introduction to Firewall Analytics - Interfirewall and Transfirewall.pptx
kuthubussaman1
 
PPTX
Introduction to Knowledge Engineering Part 1
busyprogrammersguide
 
PPTX
The THESIS FINAL-DEFENSE-PRESENTATION.pptx
hshakobe3
 
PPT
ISS -ESG Data flows What is ESG and HowHow
lucandthemachine
 
PPTX
STUDY DESIGN details- Lt Col Maksud (21).pptx
AyeshaAsha11
 
PPTX
Supervised vs unsupervised machine learning algorithms
agarwal18harsh08
 
PPTX
oil_refinery_comprehensive_20250804084928 (1).pptx
TusharPrajapati65
 
PPTX
iec ppt-1 pptx icmr ppt on rehabilitation.pptx
thirishadevan81
 
PDF
168300704-gasification-ppt.pdfhghhhsjsjhsuxush
sriram270905
 
PDF
Introduction to Data Science and Data Analysis
Suraj Patil
 
PPTX
Qualitative Qantitative and Mixed Methods.pptx
AhmaduMohammed
 
PPTX
Data_Analytics_and_PowerBI_Presentation.pptx
ZubyrAhmed
 
PPT
Miokarditis (Inflamasi pada Otot Jantung)
NandaAndini1
 
PPTX
Introduction to Basics of Ethical Hacking and Penetration Testing -Unit No. 1...
AshutoshDeshmukh33
 
PDF
Clinical guidelines as a resource for EBP(1).pdf
MashalKhan626345
 
IB Computer Science - Internal Assessment.pptx
agarwal18harsh08
 
climate analysis of Dhaka ,Banglades.pptx
rrawjatun
 
Computer network topology notes for revision
BatoolRawat
 
STERILIZATION AND DISINFECTION-1.ppthhhbx
DivuuJain
 
mbdjdhjjodule 5-1 rhfhhfjtjjhafbrhfnfbbfnb
CHINTU5000
 
Introduction to Firewall Analytics - Interfirewall and Transfirewall.pptx
kuthubussaman1
 
Introduction to Knowledge Engineering Part 1
busyprogrammersguide
 
The THESIS FINAL-DEFENSE-PRESENTATION.pptx
hshakobe3
 
ISS -ESG Data flows What is ESG and HowHow
lucandthemachine
 
STUDY DESIGN details- Lt Col Maksud (21).pptx
AyeshaAsha11
 
Supervised vs unsupervised machine learning algorithms
agarwal18harsh08
 
oil_refinery_comprehensive_20250804084928 (1).pptx
TusharPrajapati65
 
iec ppt-1 pptx icmr ppt on rehabilitation.pptx
thirishadevan81
 
168300704-gasification-ppt.pdfhghhhsjsjhsuxush
sriram270905
 
Introduction to Data Science and Data Analysis
Suraj Patil
 
Qualitative Qantitative and Mixed Methods.pptx
AhmaduMohammed
 
Data_Analytics_and_PowerBI_Presentation.pptx
ZubyrAhmed
 
Miokarditis (Inflamasi pada Otot Jantung)
NandaAndini1
 
Introduction to Basics of Ethical Hacking and Penetration Testing -Unit No. 1...
AshutoshDeshmukh33
 
Clinical guidelines as a resource for EBP(1).pdf
MashalKhan626345
 

Big data and hadoop training - Session 5

  • 1. Big Data and Hadoop Training Session 5
  • 2. Big Data - Pipeline Big Data Pipeline Lambda Architecture - Streaming(Real-Time) Layer with Apache Kafka Apache Hadoop Apache Spark Apache Cassandra on Amazon Web Services Cloud Platform
  • 3. Big Data - Pipeline
  • 4. Big Data - Pipeline 3 EC2 instance for Kafka Cluster
  • 5. Big Data - Pipeline Repeat commands for all - 3 EC2 instance for Kafka Cluster cat /etc/*-release sudo add-apt-repository ppa:webupd8team/java sudo apt-get update sudo apt-get install oracle-java8-installer java -version mkdir kafka cd kafka wget http://download.nextag.com/apache/kafka/0.10.0.0/kafka_2.11-0.10.0.0.tgz tar -zxvf kafka_2.11-0.10.0.0.tgz cd kafka_2.11-0.10.0.0 ZooKeeper ==> 172.31.48.208 / 52.91.1.93 Kafka-datanode1 ==> 172.31.63.203 / 54.173.215.211 Kafka-datanode2 ==> 172.31.9.25 / 54.226.29.194
  • 6. Big Data - Pipeline Kafka-datanode1 (set following properties for config/server.properties) ubuntu@ip-172-31-63-203:~/kafka/kafka_2.11-0.10.0.0$ vi config/server.properties broker.id=1 listeners=PLAINTEXT://172.31.63.203:9092 advertised.listeners=PLAINTEXT://54.173.215.211:9092 zookeeper.connect=52.91.1.93:2181 Kafka-datanode2 (set following properties for config/server.properties) ubuntu@ip-172-31-9-25:~/kafka/kafka_2.11-0.10.0.0$ vi config/server.properties broker.id=2 listeners=PLAINTEXT://172.31.9.25:9092 advertised.listeners=PLAINTEXT://54.226.29.194:9092 zookeeper.connect=52.91.1.93:2181 Modify config/server.properties for kafka-datanode1 & kafkadatanode2 ZooKeeper ==> 172.31.48.208 / 52.91.1.93 Kafka-datanode1 ==> 172.31.63.203 / 54.173.215.211 Kafka-datanode2 ==> 172.31.9.25 / 54.226.29.194
  • 7. Big Data - Pipeline Launch zookeeper / datanode1 / datanode2 ZooKeeper ==> 172.31.48.208 / 52.91.1.93 Kafka-datanode1 ==> 172.31.63.203 / 54.173.215.211 Kafka-datanode2 ==> 172.31.9.25 / 54.226.29.194 1) Start zookeeper bin/zookeeper-server-start.sh config/zookeeper.properties 2) Start server on Kafka-datanode1 bin/kafka-server-start.sh config/server.properties 3) Start server on Kafka-datanode2 bin/kafka-server-start.sh config/server.properties export KAFKA_HEAP_OPTS="-Xmx256M -Xms256M" 4) Create Topic & Start consumer bin/kafka-topics.sh --zookeeper 52.91.1.93:2181 --create --topic data --partitions 1 --replication-factor 2 bin/kafka-console-consumer.sh --zookeeper 52.91.1.93:2181 --topic data --from-beginning
  • 8. Big Data - Pipeline Launch Kafka Cluster (Zookeeper/kafka datanode1/ kafka datanode2)
  • 9. Big Data - Pipeline Execute Python / Kafka Spark Job
  • 10. Big Data - Pipeline Sample data which we will be sending to Kafka Server from Java Kafka Producer (csv file)
  • 11. Big Data - Pipeline Python Spark Job Processing Data from AWS Kafka Cluster
  • 12. Big Data - Pipeline Python Spark Streaming Application