Ncku csie talk about Spark
3 likes1,094 views
This document provides an introduction and agenda for a presentation on Spark. It discusses how Spark is a fast engine for large-scale data processing and how it improves on MapReduce. Spark stores data in memory across clusters to allow for faster iterative computations versus writing to disk with MapReduce. The presentation will demonstrate Spark concepts through word count and log analysis examples and provide an overview of Spark's Resilient Distributed Datasets (RDDs) and directed acyclic graph (DAG) execution model.
![Who am I?
• Wisely Chen ( thegiive@gmail.com )
• Sr. Engineer inYahoo![Taiwan] data team
• Loves to promote open source tech
• Hadoop Summit 2013 San Jose
• Jenkins Conf 2013 Palo Alto
• Spark Summit 2014 San Francisco
• Coscup 2006, 2012, 2013 , OSDC 2007,Webconf 2013,
Coscup 2012, PHPConf 2012 , RubyConf 2012](https://image.slidesharecdn.com/nckucsietalkslides-141009214416-conversion-gate01/85/Ncku-csie-talk-about-Spark-3-320.jpg)
![MLlib
• Data
• data: [(36, 2802, 4.0), (36, 256, 4.0), …]
• rank, numIter, lambda are int
• candidates : [(0, 2),(0, 3),(0, 4)…]
• model = ALS.train(data, rank, numIter, lambda)
• model.predictAll(candidates)](https://image.slidesharecdn.com/nckucsietalkslides-141009214416-conversion-gate01/85/Ncku-csie-talk-about-Spark-95-320.jpg)
Ncku csie talk about Spark
- 1. Introduction to Spark Wisely Chen (thegiive@gmail.com) Sr. Engineer at Yahoo
- 2. Agenda • Big data will change the world? • What is Spark? • Demo (Start a spark cluster / Word Count) • Break : 10min • Spark Concept • Demo (ETL / MLib) • Q&A
- 3. Who am I? • Wisely Chen ( thegiive@gmail.com ) • Sr. Engineer inYahoo![Taiwan] data team • Loves to promote open source tech • Hadoop Summit 2013 San Jose • Jenkins Conf 2013 Palo Alto • Spark Summit 2014 San Francisco • Coscup 2006, 2012, 2013 , OSDC 2007,Webconf 2013, Coscup 2012, PHPConf 2012 , RubyConf 2012
- 4. Taiwan Data Team Data! Highway BI! Report Serving! API Data! Mart ETL / Forecast Machine! Learning
- 5. Big data will change the world?
- 10. More Sensor • 2000 • Browser • Digital Camera(Photo) • 2000~2014 • Browser • Mobile (GPS,More Photo,Video) • Wearable Device(Pulse) • Google Glass(More Video) • Internet of thing(………)
- 12. More Sensor • 2000 • Browser • Digital Camera(Photo) • 2000~2014 • Browser • Mobile (GPS,More Photo,Video) • Wearable Device(Pulse) • Google Glass(More Video) • Internet of thing(………)
- 13. Technology Improve • Sloan Digital Sky Survey(SDSS) collected more data in its first few weeks than had been amassed in the entire history of astronomy. • The Large Synoptic Survey Telescope in Chile, due to come on stream in 2016, will acquire that quantity of data every five days.
- 14. New Area 30min Zebra fish experiment = 1TB http://research.janelia.org/zebrafish/
- 15. Hadoop handle big data well • 18M of hadoop related jobs on Yahoo Grid • Yahoo handle over 440 PB data daily • Most of job are ETL/SQL/BI
- 16. EBay’s data volume 2015 : 130EB 2020 : 4000ZB Vadim Kutsyy “Data Science Empowering Personalization” in Big data innovation Summit 2014 Boston
- 17. Data is not only bigger • We have more area of data • More Sensor • Sensor technology improve • New area
- 19. Word Grammar Check • MS researcher Michele and Eric try to improve grammar check algorithm • They took four algorithm and feed in 10M, 100M and 1B words • In 10M words, sophisticated algorithm(86%) works bester than simpler algorithm(75%) • In 1B words, simpler algorithm(95%+) improved a lot, even better than sophisticated algorithm(94%)
- 20. –Google AI guru Peter Norvig, "The Unreasonable Effectiveness of Data” “Simple models and a lot of data trump more elaborate models based on less data” In translate area
- 21. Different type of data • In Harvard Data Mining Class, two team do Netflix recommendation challenge • Team A came up with a very sophisticated algorithm using the Netflix data. • Team B used a very simple algorithm, but they added in additional data beyond the Netflix set • Team B got much better results, close to the best results on the Netflix leaderboard
- 22. Taiwan Shopping User Analysis Man tend to view underwear. But they don’t buy it Male Users’ Top5 View Categories 1. Computer 2. Camera 3. ………. 4. ………. 5. Woman Underwear
- 23. 2 types of data Traffic Data Transaction Data User’s views / clicks “Weak intention” Large amount User’s checkout “Strong Intention” Small amour
- 24. Small Data Sophisticated Algo OK Result Big Data Simple Algo Better Result Data Set 1 Smart Model which leverage more area of data Data Set 2 Best Result
- 26. Robot • FoxCon’s robot can replace 70% worker • Google driverless car/big dog • Amazon warehouse robot
- 28. It is not movie it is happening
- 29. Sensor Data Machine Learning Robot User Behavior Recommendation Algorithm Recommendation to user (1/3 sales are from recommendation module) Amazon layoff the editor team and replaced by recommendation algorithm
- 31. Sensor Data Machine Learning Robot DNNresearch, Behavio, Wavii, Flutter, autofuss, DeepMind, spider.io, Adometry, QQuest Visual, Jetpac Talaria, Stackdriver SCHAFT, Industrial Perception, Redwood Robotics, Meka Robotics, Holomni, Bot & Dolly, Boston Dynamics, Titan Aerospace, Nest Lab, MyEnergy, Skybox Imaging, Dropcam, Google buy 47 company at 13,14 IOT Google is top 1 leader in big data 24 company on the ring
- 34. The ring will change the world and big data is the core of ring
- 37. Hadoop is not good in machine learning • Efficiency • Difficult • Data Engineer • Data Scientist • Data Analyst • Algorithm: it is not so easy to parallelize your algorithm
- 39. Hadoop is not good in machine learning • Efficiency • Difficult : Data scientist don’t know how to do • Algorithm: it is not so easy to parallelize your algorithm
- 40. 3X~25X than MapReduce framework ! From Matei’s paper: http://0rz.tw/VVqgP Logistic regression RunningTime(S) 0 20 40 60 80 MR Spark 3 76 KMeans 0 27.5 55 82.5 110 MR Spark 33 106 PageRank 0 45 90 135 180 MR Spark 23 171
- 41. Hadoop is not good in machine learning • Efficiency • Difficult • Algorithm: it is not so easy to parallelize your algorithm
- 43. Language Support • Python : Data Scientist , Data Engineer • Java : Data Engineer • Scala : Data Engineer • SQL : Data Scientist, Data Analyst , Data Engineer • R : Data Scientist, Data Analyst • (will be official support in 1.2)
- 44. Python Word Count • file = spark.textFile("hdfs://...") • counts = file.flatMap(lambda line: line.split(" ")) • .map(lambda word: (word, 1)) • .reduceByKey(lambda a, b: a + b) • counts.saveAsTextFile("hdfs://...") Access data via Spark API Process via Python
- 45. Scala Word Count • val file = spark.textFile("hdfs://...") • val counts = file.flatMap(line => line.split(" ")) • .map(word => (word, 1)) • .reduceByKey(_ + _) • counts.saveAsTextFile("hdfs://...")
- 46. Java Wordcount • JavaRDD<String> file = spark.textFile("hdfs://..."); • JavaRDD<String> words = file.flatMap(new FlatMapFunction<String, String>() • public Iterable<String> call(String s) { return Arrays.asList(s.split(" ")); } • }); • JavaPairRDD<String, Integer> pairs = words.map(new PairFunction<String, String, Integer>() • public Tuple2<String, Integer> call(String s) { return new Tuple2<String, Integer>(s, 1); } • }); • JavaPairRDD<String, Integer> counts = pairs.reduceByKey(new Function2<Integer, Integer>() • public Integer call(Integer a, Integer b) { return a + b; } • }); • counts.saveAsTextFile("hdfs://...");
- 47. Highly Recommend • Scala : Latest API feature, Stable • Python • very familiar language • Native Lib: NumPy, SciPy
- 48. What is Spark • From UC Berkeley AMP Lab • Apache Spark™ is a very fast and general engine for large-scale data processing • Most activity Big data open source project since Hadoop
- 49. Community
- 50. Where is Spark?
- 51. HDFS YARN MapReduce Hadoop 2.0 Storm HBase Others
- 53. HDFS YARN MapReduce Hadoop vs Spark Spark Hive Shark/SparkSQL
- 54. More than MapReduce HDFS Spark Core : MapReduce SparkSQL: Hive GraphX: Pregel MLib: Mahout Streaming: Storm Resource Management System(Yarn, Mesos)
- 55. How to use it? • 1. go to https://spark.apache.org/ • 2. Download and unzip it • 3. ./sbin/start-all.sh or ./bin/spark-shell
- 56. EC2 • ./ec2/spark-ec2 -k xxx -i xxx -s 3 launch CLUSTERNAME ! • http://spark.apache.org/docs/latest/ec2- scripts.html
- 57. DEMO
- 58. Python Word Count • file = spark.textFile("hdfs://...") • counts = file.flatMap(lambda line: line.split(" ")) • .map(lambda word: (word, 1)) • .reduceByKey(lambda a, b: a + b) • counts.saveAsTextFile("hdfs://...")
- 59. BREAK
- 60. Spark Concept
- 61. Why is Spark so fast?
- 62. Most machine learning algorithms need iterative computing
- 63. a1.0 1.0 1.0 1.0 PageRank 1st Iter 2nd Iter 3rd Iter b d c Rank Tmp Result Rank Tmp Result a1.85 1.0 0.58 b d c 0.58 a1.31 1.72 0.39 b d c 0.58
- 64. HDFS is 100x slower than memory Input (HDFS) Iter 1 Tmp (HDFS) Iter 2 Tmp (HDFS) Iter N Input (HDFS) Iter 1 Tmp (Mem) Iter 2 Tmp (Mem) Iter N MapReduce Spark
- 65. First iteration(HDFS)! take 200 sec 3rd iteration(mem)! take 7.7 sec Page Rank algorithm in 1 billion record url 2nd iteration(mem)! take 7.4 sec
- 66. Memory Size Problem Cache storage in local disk(2sec) Cache storage in memory(2sec) Network transfer(30 sec)
- 67. Just memory? • From Matei’s paper: http://0rz.tw/VVqgP • HBM: stores data in an in-memory HDFS instance. • SP : Spark • HBM’1, SP’1 : first run • Storage: HDFS with 256 MB blocks • Node information • m1.xlarge EC2 nodes • 4 cores • 15 GB of RAM
- 68. 100GB data on 100 node cluster Logistic regression RunningTime(S) 0 35 70 105 140 HBM'1 HBM SP'1 SP 3 46 62 139 KMeans RunningTime(S) 0 50 100 150 200 HBM'1 HBM SP'1 SP 33 8287 182
- 71. map,filter groupBy on ! non-partitioned data union join with input! co-partitioned join with inputs not! co-partitioned Map(Narrow) Reduce(Wide)
- 77. RDD • Resilient Distributed Dataset • Interface of data, stored in RAM or on Disk • Built through parallel transformations
- 78. RDD RDD a RDD b val a =sc.textFile(“hdfs://....”) val b = a.filer( line=>line.contain(“Spark”) ) Value c val c = b.count() Transformation Action
- 79. Log mining a = sc.textfile(“hdfs://aaa.com/a.txt”)! err = a.filter(lambda t=> “ERROR” in t )! .filter(lambda t=> “2014” in t)! ! err.cache()! err.count()! ! m = err.filter(lambda t=>“MYSQL” in t)! ! ! .count()! a = err.filter(lambda t=> “APACHE” in t )! ! ! .count() Driver Worker! ! ! ! Worker! ! ! ! Worker! ! ! !Task TaskTask
- 80. Log mining a = sc.textfile(“hdfs://aaa.com/a.txt”)! err = a.filter(lambda t=> “ERROR” in t )! .filter(lambda t=> “2014” in t)! ! err.cache()! err.count()! ! m = err.filter(lambda t=>“MYSQL” in t)! ! ! .count()! a = err.filter(lambda t=> “APACHE” in t )! ! ! .count() Driver Worker! ! ! ! !Block1 RDD a Worker! ! ! ! !Block2 RDD a Worker! ! ! ! !Block3 RDD a
- 81. Log mining a = sc.textfile(“hdfs://aaa.com/a.txt”)! err = a.filter(lambda t=> “ERROR” in t )! .filter(lambda t=> “2014” in t)! ! err.cache()! err.count()! ! m = err.filter(lambda t=>“MYSQL” in t)! ! ! .count()! a = err.filter(lambda t=> “APACHE” in t )! ! ! .count() Driver Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Block1 Block2 Block3
- 82. Log mining a = sc.textfile(“hdfs://aaa.com/a.txt”)! err = a.filter(lambda t=> “ERROR” in t )! .filter(lambda t=> “2014” in t)! ! err.cache()! err.count()! ! m = err.filter(lambda t=>“MYSQL” in t)! ! ! .count()! a = err.filter(lambda t=> “APACHE” in t )! ! ! .count() Driver Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Block1 Block2 Block3
- 83. Log mining a = sc.textfile(“hdfs://aaa.com/a.txt”)! err = a.filter(lambda t=> “ERROR” in t )! .filter(lambda t=> “2014” in t)! ! err.cache()! err.count()! ! m = err.filter(lambda t=>“MYSQL” in t)! ! ! .count()! a = err.filter(lambda t=> “APACHE” in t )! ! ! .count() Driver Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Cache1 Cache2 Cache3
- 84. Log mining a = sc.textfile(“hdfs://aaa.com/a.txt”)! err = a.filter(lambda t=> “ERROR” in t )! .filter(lambda t=> “2014” in t)! ! err.cache()! err.count()! ! m = err.filter(lambda t=>“MYSQL” in t)! ! ! .count()! a = err.filter(lambda t=> “APACHE” in t )! ! ! .count() Driver Worker! ! ! ! ! RDD m Worker! ! ! ! ! RDD m Worker! ! ! ! ! RDD m Cache1 Cache2 Cache3
- 85. Log mining a = sc.textfile(“hdfs://aaa.com/a.txt”)! err = a.filter(lambda t=> “ERROR” in t )! .filter(lambda t=> “2014” in t)! ! err.cache()! err.count()! ! m = err.filter(lambda t=>“MYSQL” in t)! ! ! .count()! a = err.filter(lambda t=> “APACHE” in t )! ! ! .count() Driver Worker! ! ! ! ! RDD a Worker! ! ! ! ! RDD a Worker! ! ! ! ! RDD a Cache1 Cache2 Cache3
- 86. 1st iteration(no cache)! take same time with cache! take 7 sec RDD Cache
- 87. RDD Cache • Data locality • Cache A big shuffle! take 20min After cache, take only 265ms self join 5 billion record data
- 88. DEMO
- 89. Log Mining
- 90. Page Rank
- 91. a1.0 1.0 1.0 1.0 PageRank 1st Iter 2nd Iter 3rd Iter b d c Rank Tmp Result Rank Tmp Result a1.85 1.0 0.58 b d c 0.58 a1.31 1.72 0.39 b d c 0.58
- 92. SparkSQL
- 93. Recommendation
- 95. MLlib • Data • data: [(36, 2802, 4.0), (36, 256, 4.0), …] • rank, numIter, lambda are int • candidates : [(0, 2),(0, 3),(0, 4)…] • model = ALS.train(data, rank, numIter, lambda) • model.predictAll(candidates)
- 96. Homework • 1. Install Spark and run word count (50%) • Data : http://www.gutenberg.org/ebooks/5000 • Output: total word number • 2. Write Movie Recommendation (50%) • Trainning Data : http://arbor.ee.ntu.edu.tw/~wisely/data/lesson.tgz • Input: 10 rating(1-5) on the 10 movie • Example: movie 123 rating is 3 , movie 45 is 5 • Output: Top 10 recommendation movie • Any algorithm is ok
- 99. Background Knowledge • Tweet real time data store into SQL database • Spark MLLib use Wikipedia data to train a TF- IDF model • SparkSQL select tweet and filter by TF-IDF model • Generate live BI report
- 100. Code • val wiki = sql(“select text from wiki”) • val model = new TFIDF() • model.train(wiki) • registerFunction(“similarity” , model.similarity _ ) • select tweet from tweet where similarity(tweet, “$search” > 0.01 )
- 102. Q & A