Big data school demo
Download as PPTX, PDF0 likes477 views
This document outlines an agenda for a Big Data demo covering setup of Spark, Kafka, and Zeppelin; an ETL example to move Twitter data to Kafka using Spark; data processing and visualization with Spark and Zeppelin; and a machine learning example using Spark MLLib. It provides instructions for installing and configuring the technologies, developing a simple ETL pipeline, and accessing an example Zeppelin notebook.
Big data school demo
- 2. Agenda ○ Setting up environment (Spark + Kafka + Zeppelin) ○ ETL example: Twitter -> Kafka ○ Data processing with Spark ○ Data visualization with Zeppelin ○ Spark MLLib: simple machine learning example
- 3. Setup Spark Download Spark 1.6.3 from spark.apache.org/downloads.html > tar xzf spark-1.6.3-bin-hadoop2.6.tgz > cd spark-1.6.3-bin-hadoop2.6 Start Master & Slave > ./sbin/start-master.sh > ./sbin/start-slave.sh spark://Aleksandrs-MacBook-Pro.local:7077
- 4. Setup Spark
- 5. Check Spark is working > ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://Aleksandrs-MacBook-Pro.local:7077 lib/spark-examples-1.6.3-hadoop2.6.0.jar 10 Find at log: … Pi is roughly 3.1395271395271394 ...
- 6. Check Spark is working
- 7. Kafka setup Download the 0.8.2.1 release and un-tar it. > tar -xzf kafka_2.10-0.8.2.1.tgz > cd kafka_2.10-0.8.2.1 Start Server > nohup bin/zookeeper-server-start.sh config/zookeeper.properties & > nohup bin/kafka-server-start.sh config/server.properties &
- 8. Kafka setup Create topic > bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 -- partitions 1 --topic test > bin/kafka-topics.sh --list --zookeeper localhost:2181 Run consumer > bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from- beginning
- 9. Start simple ETL > ./bin/spark-submit --class com.dataart.bigdataschool.SimpleETLExample --master spark://Aleksandrs-MacBook-Pro.local:7077 --deploy-mode cluster --total-executor-cores 2 /Users/apavlenko/github/BigDataSchoolDemo/SimpleETLExample/target/scala- 2.10/SimpleETLExample-assembly-0.1.0.jar localhost:9092 test Simple ETL Example Gist: https://gist.github.com/AleksandrPavlenko/897c930918bc079048774df170d88085
- 10. Zeppelin setup Download and un-tar it https://zeppelin.apache.org/download.html > cp conf/zeppelin-site.xml.template conf/zeppelin-site.xml > cp conf/zeppelin-env.sh.template conf/zeppelin-env.sh Change zeppelin.server.port to 8090 in zeppelin-site.xml Set SPARK_HOME in zeppelin-env.sh Run Zeppelin > ./bin/zeppelin-daemon.sh start
- 12. Big Data School Zeppelin notebook https://github.com/AleksandrPavlenko/BigDataSchoolKharkiv