SlideShare a Scribd company logo

OCF.tw's talk about "Introduction to spark"

Giivee The, profile picture
Giivee The

The document is a presentation by Wisely Chen, a Senior Engineer at Yahoo, introducing Apache Spark, a fast engine for large-scale data processing that is seen as the successor to MapReduce. It covers the Spark ecosystem, its advantages over Hadoop, and provides examples of using Spark with Python, Java, and Scala for data processing tasks. The presentation also highlights Spark's real-time analytics capabilities, machine learning applications, and the future of data processing frameworks.

Technology
1 of 80
Downloaded 103 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
Introduction to Spark Wisely Chen (aka thegiive) Sr. Engineer at Yahoo
Agenda • What is Spark? ( Easy ) • Spark Concept ( Middle ) • Break : 10min • Spark EcoSystem ( Easy ) • Spark Future ( Middle ) • Q&A
Who am I? • Wisely Chen ( thegiive@gmail.com ) • Sr. Engineer in Yahoo![Taiwan] data team • Loves to promote open source tech • Hadoop Summit 2013 San Jose • Jenkins Conf 2013 Palo Alto • Spark Summit 2014 San Francisco • Coscup 2006, 2012, 2013 , OSDC 2007, Webconf 2013, Coscup 2012, PHPConf 2012 , RubyConf 2012
Taiwan Data Team Data! Highway BI! Report Serving! API Data! Mart ETL / Forecast Machine! Learning
OCF.tw's talk about "Introduction to spark"
Forecast Recommendation
HADOOP
Opinion from Cloudera • The leading candidate for “successor to MapReduce” today is Apache Spark • No vendor — no new project — is likely to catch up. Chasing Spark would be a waste of time, and would delay availability of real-time analytic and processing services for no good reason. ! • From http://0rz.tw/y3OfM
What is Spark • From UC Berkeley AMP Lab • Most activity Big data open source project since Hadoop
Community
Community
Where is Spark?
YARN HDFS MapReduce Hadoop 2.0 Storm HBase Others
Hadoop Architecture Hive MapReduce YARN HDFS SQL Computing Engine Resource Management Storage
Hadoop vs Spark Hive Shark/SparkSQL YARN HDFS MapReduce Spark
Spark vs Hadoop • Spark run on Yarn, Mesos or Standalone mode • Spark’s main concept is based on MapReduce • Spark can read from • HDFS: data locality • HBase • Cassandra
More than MapReduce Shark: Hive GraphX: Pregel MLib: Mahout Spark Core : MapReduce HDFS Streaming: Storm Resource Management System(Yarn, Mesos)
Why Spark?
天下武功，無堅不破，惟快不破
Logistic regression 3 110 82.5 55 27.5 33 106 180 135 90 45 171 3X~25X than MapReduce framework ! From Matei’s paper: http://0rz.tw/VVqgP Running Time(S) 80 60 40 20 0 76 MR Spark KMeans 0 MR Spark PageRank 0 23 MR Spark
What is Spark • Apache Spark™ is a very fast and general engine for large-scale data processing
Language Support • Python • Java • Scala
Python Word Count • file = spark.textFile("hdfs://...") • counts = file.flatMap(lambda line: line.split(" ")) • .map(lambda word: (word, 1)) • .reduceByKey(lambda a, b: a + b) • counts.saveAsTextFile("hdfs://...") Access data via Spark API Process via Python
What is Spark • Apache Spark™ is a very fast and general engine for large-scale data processing
Why is Spark so fast?
Most machine learning algorithms need iterative computing
a 1.0 1.0 1.0 1.0 PageRank b b 1st Iter 2nd Iter 3rd Iter b d c Rank Tmp Result Rank Tmp Result a 1.85 1.0 0.58 d c 0.58 a 1.31 1.72 0.39 d c 0.58
HDFS is 100x slower than memory Input (HDFS) Iter 1 Tmp (HDFS) Iter 2 Tmp (HDFS) Iter N Input (HDFS) Iter 1 Tmp (Mem) Iter 2 Tmp (Mem) Iter N MapReduce Spark
3rd iteration(mem)! take 7.7 sec 2nd iteration(mem)! take 7.4 sec First iteration(HDFS)! take 200 sec Page Rank algorithm in 1 billion record url
Spark Concept
Map Reduce Shuffle
DAG Engine
DAG Engine
RDD • Resilient Distributed Dataset • Collections of objects spread across a cluster, stored in RAM or on Disk • Built through parallel transformations
Fault Tolerance 天下武功，無堅不破，惟快不破
RDD val b = a.filer( line=>line.contain(“Spark”) ) RDD a RDD b val a =sc.textFile(“hdfs://....”) Value c val c = b.count() Transformation Action
Log mining val a = sc.textfile(“hdfs://aaa.com/a.txt”)! val err = a.filter( t=> t.contains(“ERROR”) )! .filter( t=>t.contains(“2014”)! ! err.cache()! err.count()! ! val m = err.filter( t=> t.contains(“MYSQL”) )! ! ! .count()! val a = err.filter( t=> t.contains(“APACHE”) )! ! ! .count() Driver Worker! ! ! ! Worker! ! ! Tas! k Worker! ! ! ! Task Task
Log mining val a = sc.textfile(“hdfs://aaa.com/a.txt”)! val err = a.filter( t=> t.contains(“ERROR”) )! .filter( t=>t.contains(“2014”)! ! err.cache()! err.count()! ! val m = err.filter( t=> t.contains(“MYSQL”) )! ! ! .count()! val a = err.filter( t=> t.contains(“APACHE”) )! ! ! .count() Driver Worker! ! ! ! RDD a Bloc! k1 Worker! ! ! ! RDD a Bloc! k3 Worker! ! ! ! RDD a Bloc! k2
Log mining val a = sc.textfile(“hdfs://aaa.com/a.txt”)! val err = a.filter( t=> t.contains(“ERROR”) )! .filter( t=>t.contains(“2014”)! ! err.cache()! err.count()! ! val m = err.filter( t=> t.contains(“MYSQL”) )! ! ! .count()! val a = err.filter( t=> t.contains(“APACHE”) )! ! ! .count() Driver Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Block3 Worker! ! ! ! ! RDD err Block1 Block2
Log mining val a = sc.textfile(“hdfs://aaa.com/a.txt”)! val err = a.filter( t=> t.contains(“ERROR”) )! .filter( t=>t.contains(“2014”)! ! err.cache()! err.count()! ! val m = err.filter( t=> t.contains(“MYSQL”) )! ! ! .count()! val a = err.filter( t=> t.contains(“APACHE”) )! ! ! .count() Driver Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Block3 Worker! ! ! ! ! RDD err Block1 Block2
Log mining val a = sc.textfile(“hdfs://aaa.com/a.txt”)! val err = a.filter( t=> t.contains(“ERROR”) )! .filter( t=>t.contains(“2014”)! ! err.cache()! err.count()! ! val m = err.filter( t=> t.contains(“MYSQL”) )! ! ! .count()! val a = err.filter( t=> t.contains(“APACHE”) )! ! ! .count() Driver Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Cache3 Worker! ! ! ! ! RDD err Cache1 Cache2
Log mining val a = sc.textfile(“hdfs://aaa.com/a.txt”)! val err = a.filter( t=> t.contains(“ERROR”) )! .filter( t=>t.contains(“2014”)! ! err.cache()! err.count()! ! val m = err.filter( t=> t.contains(“MYSQL”) )! ! ! .count()! val a = err.filter( t=> t.contains(“APACHE”) )! ! ! .count() Driver Worker! ! ! ! ! RDD m Worker! ! ! ! ! RDD m Cache3 Worker! ! ! ! ! RDD m Cache1 Cache2
Log mining val a = sc.textfile(“hdfs://aaa.com/a.txt”)! val err = a.filter( t=> t.contains(“ERROR”) )! .filter( t=>t.contains(“2014”)! ! err.cache()! err.count()! ! val m = err.filter( t=> t.contains(“MYSQL”) )! ! ! .count()! val a = err.filter( t=> t.contains(“APACHE”) )! ! ! .count() Driver Worker! ! ! ! ! RDD a Worker! ! ! ! ! RDD a Cache3 Worker! ! ! ! ! RDD a Cache1 Cache2
RDD Cache with cache! take 7 sec 1st iteration(no cache)! take same time
RDD Cache • Data locality • Cache After cache, take only 265ms A big shuffle! take 20min self join 5 billion record data
Scala Word Count • val file = spark.textFile("hdfs://...") • val counts = file.flatMap(line => line.split(" ")) • .map(word => (word, 1)) • .reduceByKey(_ + _) • counts.saveAsTextFile("hdfs://...")
Step by Step • file.flatMap(line => line.split(" “)) => (aaa,bb,cc) • .map(word => (word, 1)) => ((aaa,1),(bb,1)..) • .reduceByKey(_ + _) => ((aaa,123),(bb,23)…)
Java Wordcount • JavaRDD<String> file = spark.textFile("hdfs://..."); • JavaRDD<String> words = file.flatMap(new FlatMapFunction<String, String>() • public Iterable<String> call(String s) { return Arrays.asList(s.split(" ")); } • }); • JavaPairRDD<String, Integer> pairs = words.map(new PairFunction<String, String, Integer>() • public Tuple2<String, Integer> call(String s) { return new Tuple2<String, Integer>(s, 1); } • }); • JavaPairRDD<String, Integer> counts = pairs.reduceByKey(new Function2<Integer, Integer>() • public Integer call(Integer a, Integer b) { return a + b; } • }); • counts.saveAsTextFile("hdfs://...");
Java vs Scala • Scala : file.flatMap(line => line.split(" ")) • Java version : • JavaRDD<String> words = file.flatMap(new FlatMapFunction<String, String>() • public Iterable<String> call(String s) { • return Arrays.asList(s.split(" ")); } • });
Python • file = spark.textFile("hdfs://...") • counts = file.flatMap(lambda line: line.split(" ")) • .map(lambda word: (word, 1)) • .reduceByKey(lambda a, b: a + b) • counts.saveAsTextFile("hdfs://...")
Highly Recommend • Scala : Latest API feature, Stable • Python • very familiar language • Native Lib: NumPy, SciPy
How to use it? • 1. go to https://spark.apache.org/ • 2. Download and unzip it • 3. ./sbin/start-all.sh or ./bin/spark-shell
DEMO
EcoSystem/Future
OCF.tw's talk about "Introduction to spark"
Hadoop EcoSystem
Hadoop EcoSystem
Spark ECOSystem SparkSQL: Hive GraphX: Pregel MLib: Mahout Spark Core : MapReduce HDFS Streaming: Storm Resource Management System(Yarn, Mesos)
Unified Platform
Detail Streaming BI ETL Spark SparkSQL MLlib Hive HDFS Cassandra RDBMS
Complexity
Performance
Write once, Run use case
BI (SparkSQL) Streaming (SparkStreaming) Machine Learning (MLlib) Spark
Spark bridge people together
Data Analyst Data Engineer Data Scientist
Bridge people together • Scala : Engineer • Java : Engineer • Python : Data Scientist , Engineer • R : Data Scientist , Data Analyst • SQL : Data Analyst
Yahoo EC team Data Platform! ! ! ! ! ! ! ! ! ! Filtered Data! (HDFS) Data Mart! (Oracle) ML Model! (Spark) BI Report! (MSTR) Traffic! Data Transaction! Data Shark
Data Analyst
Data Analyst 350 TB data • Select tweet from tweets_data where similarity(tweet , “FIFA” ) > 0.01 Machine ! Learning • = ! • http://youtu.be/lO7LhVZrNwA?list=PL-x35fyliRwiST9gF7Z8Nu3LgJDFRuwfr https://www.youtube.com/watch?v=lO7LhVZrNwA&list=PL-x35fyliRwiST9gF7Z8Nu3LgJDFRuwfr#t=2900
Data Scientist http://goo.gl/q5CAx8 http://research.janelia.org/zebrafish/
SQL (Data Analyst) Cloud Computing (Data Engineer) Machine Learning (Data Scientist) Spark
Databricks Cloud DEMO
BI (SparkSQL) Streaming (SparkStreaming) Machine Learning (MLlib) Spark
Instant BI Report http://youtu.be/dJQ5lV5Tldw?t=30m30s
BI (SparkSQL) Streaming (SparkStreaming) Machine Learning (MLlib) Spark
Background Knowledge • Tweet real time data store into SQL database • Spark MLLib use Wikipedia data to train a TF-IDF model • SparkSQL select tweet and filter by TF-IDF model • Generate live BI report
Code • val wiki = sql(“select text from wiki”) • val model = new TFIDF() • model.train(wiki) • registerFunction(“similarity” , model.similarity _ ) • select tweet from tweet where similarity(tweet, “$search” > 0.01 )
DEMO http://youtu.be/dJQ5lV5Tldw?t=39m30s
Q & A

More Related Content

PDF
Big data analytics with Spark & Cassandra
Matthias Niehoff
 
PDF
Spark cassandra integration, theory and practice
Duyhai Doan
 
PDF
Zero to Streaming: Spark and Cassandra
Russell Spitzer
 
PDF
Apache Drill: Building Highly Flexible, High Performance Query Engines by M.C...
The Hive
 
PDF
Spark cassandra connector.API, Best Practices and Use-Cases
Duyhai Doan
 
PDF
Lightning fast analytics with Spark and Cassandra
nickmbailey
 
PPTX
Putting Apache Drill into Production
MapR Technologies
 
PPTX
SQL-on-Hadoop with Apache Drill
MapR Technologies
 
Big data analytics with Spark & Cassandra
Matthias Niehoff
 
Spark cassandra integration, theory and practice
Duyhai Doan
 
Zero to Streaming: Spark and Cassandra
Russell Spitzer
 
Apache Drill: Building Highly Flexible, High Performance Query Engines by M.C...
The Hive
 
Spark cassandra connector.API, Best Practices and Use-Cases
Duyhai Doan
 
Lightning fast analytics with Spark and Cassandra
nickmbailey
 
Putting Apache Drill into Production
MapR Technologies
 
SQL-on-Hadoop with Apache Drill
MapR Technologies
 

What's hot (20)

PPTX
Analytics with Cassandra, Spark & MLLib - Cassandra Essentials Day
Matthias Niehoff
 
PPTX
Working with Delimited Data in Apache Drill 1.6.0
Vince Gonzalez
 
PDF
Apache Drill @ PJUG, Jan 15, 2013
Gera Shegalov
 
PDF
Spark Cassandra Connector: Past, Present, and Future
Russell Spitzer
 
PDF
Killing ETL with Apache Drill
Charles Givre
 
PPTX
M7 and Apache Drill, Micheal Hausenblas
Modern Data Stack France
 
PDF
Spark Streaming with Cassandra
Jacek Lewandowski
 
PDF
Hadoop2 new and noteworthy SNIA conf
Sujee Maniyam
 
PDF
Intro to py spark (and cassandra)
Jon Haddad
 
PDF
A deeper-understanding-of-spark-internals
Cheng Min Chi
 
PDF
Scalding - the not-so-basics @ ScalaDays 2014
Konrad Malawski
 
PDF
Hadoop Jungle
Alexey Zinoviev
 
PDF
Swiss Big Data User Group - Introduction to Apache Drill
MapR Technologies
 
PPTX
Hadoop and Spark for the SAS Developer
DataWorks Summit
 
PDF
Open Source Logging and Metric Tools
Phase2
 
PDF
Apache Spark with Scala
Fernando Rodriguez
 
PDF
Intro To Cascading
Nate Murray
 
DOCX
Apache Drill with Oracle, Hive and HBase
Nag Arvind Gudiseva
 
PDF
Cassandra and Spark, closing the gap between no sql and analytics codemotio...
Duyhai Doan
 
PDF
Apache Spark Introduction | Big Data Hadoop Spark Tutorial | CloudxLab
CloudxLab
 
Analytics with Cassandra, Spark & MLLib - Cassandra Essentials Day
Matthias Niehoff
 
Working with Delimited Data in Apache Drill 1.6.0
Vince Gonzalez
 
Apache Drill @ PJUG, Jan 15, 2013
Gera Shegalov
 
Spark Cassandra Connector: Past, Present, and Future
Russell Spitzer
 
Killing ETL with Apache Drill
Charles Givre
 
M7 and Apache Drill, Micheal Hausenblas
Modern Data Stack France
 
Spark Streaming with Cassandra
Jacek Lewandowski
 
Hadoop2 new and noteworthy SNIA conf
Sujee Maniyam
 
Intro to py spark (and cassandra)
Jon Haddad
 
A deeper-understanding-of-spark-internals
Cheng Min Chi
 
Scalding - the not-so-basics @ ScalaDays 2014
Konrad Malawski
 
Hadoop Jungle
Alexey Zinoviev
 
Swiss Big Data User Group - Introduction to Apache Drill
MapR Technologies
 
Hadoop and Spark for the SAS Developer
DataWorks Summit
 
Open Source Logging and Metric Tools
Phase2
 
Apache Spark with Scala
Fernando Rodriguez
 
Intro To Cascading
Nate Murray
 
Apache Drill with Oracle, Hive and HBase
Nag Arvind Gudiseva
 
Cassandra and Spark, closing the gap between no sql and analytics codemotio...
Duyhai Doan
 
Apache Spark Introduction | Big Data Hadoop Spark Tutorial | CloudxLab
CloudxLab
 
Ad

Viewers also liked (7)

PPTX
Introduction to Apache Spark
Mohamed hedi Abidi
 
PDF
Introduction to Apache Spark
Samy Dindane
 
PDF
Fully fault tolerant real time data pipeline with docker and mesos
Rahul Kumar
 
PDF
An introduction To Apache Spark
Amir Sedighi
 
PPTX
Introduction to Apache Spark and MLlib
pumaranikar
 
PPT
Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive
Sachin Aggarwal
 
PDF
Introduction to Apache Spark
datamantra
 
Introduction to Apache Spark
Mohamed hedi Abidi
 
Introduction to Apache Spark
Samy Dindane
 
Fully fault tolerant real time data pipeline with docker and mesos
Rahul Kumar
 
An introduction To Apache Spark
Amir Sedighi
 
Introduction to Apache Spark and MLlib
pumaranikar
 
Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive
Sachin Aggarwal
 
Introduction to Apache Spark
datamantra
 
Ad

Similar to OCF.tw's talk about "Introduction to spark" (20)

PDF
Osd ctw spark
Wisely chen
 
PDF
xPatterns on Spark, Tachyon and Mesos - Bucharest meetup
Radu Chilom
 
PDF
Brief Intro to Apache Spark @ Stanford ICME
Paco Nathan
 
PDF
20170126 big data processing
Vienna Data Science Group
 
PDF
#MesosCon 2014: Spark on Mesos
Paco Nathan
 
PPTX
The Fundamentals Guide to HDP and HDInsight
Gert Drapers
 
PPTX
ETL with SPARK - First Spark London meetup
Rafal Kwasny
 
PDF
Cassandra Summit 2014: Apache Spark - The SDK for All Big Data Platforms
DataStax Academy
 
PDF
Apache Spark Overview @ ferret
Andrii Gakhov
 
PDF
Introduction to Apache Flink - Fast and reliable big data processing
Till Rohrmann
 
PPTX
20130912 YTC_Reynold Xin_Spark and Shark
YahooTechConference
 
PPT
Apache spark-melbourne-april-2015-meetup
Ned Shawa
 
PDF
Introduction to Apache Spark
Anastasios Skarlatidis
 
PDF
Unified Big Data Processing with Apache Spark
C4Media
 
PPTX
Apache Spark - Las Vegas Big Data Meetup Dec 3rd 2014
cdmaxime
 
PDF
Intro to apache spark stand ford
Thu Hiền
 
PDF
Introduction to Spark
Li Ming Tsai
 
PPTX
Big data, just an introduction to Hadoop and Scripting Languages
Corley S.r.l.
 
PDF
Unified Big Data Processing with Apache Spark (QCON 2014)
Databricks
 
PPTX
Paris Data Geek - Spark Streaming
Djamel Zouaoui
 
Osd ctw spark
Wisely chen
 
xPatterns on Spark, Tachyon and Mesos - Bucharest meetup
Radu Chilom
 
Brief Intro to Apache Spark @ Stanford ICME
Paco Nathan
 
20170126 big data processing
Vienna Data Science Group
 
#MesosCon 2014: Spark on Mesos
Paco Nathan
 
The Fundamentals Guide to HDP and HDInsight
Gert Drapers
 
ETL with SPARK - First Spark London meetup
Rafal Kwasny
 
Cassandra Summit 2014: Apache Spark - The SDK for All Big Data Platforms
DataStax Academy
 
Apache Spark Overview @ ferret
Andrii Gakhov
 
Introduction to Apache Flink - Fast and reliable big data processing
Till Rohrmann
 
20130912 YTC_Reynold Xin_Spark and Shark
YahooTechConference
 
Apache spark-melbourne-april-2015-meetup
Ned Shawa
 
Introduction to Apache Spark
Anastasios Skarlatidis
 
Unified Big Data Processing with Apache Spark
C4Media
 
Apache Spark - Las Vegas Big Data Meetup Dec 3rd 2014
cdmaxime
 
Intro to apache spark stand ford
Thu Hiền
 
Introduction to Spark
Li Ming Tsai
 
Big data, just an introduction to Hadoop and Scripting Languages
Corley S.r.l.
 
Unified Big Data Processing with Apache Spark (QCON 2014)
Databricks
 
Paris Data Geek - Spark Streaming
Djamel Zouaoui
 

Recently uploaded (20)

PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
PDF
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PDF
[발표본] 너의 과제는 클라우드에 있어_KTDS_김동현_20250524.pdf
ssuserf8b8bd1
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PDF
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
PPTX
breach-and-attack-simulation-cybersecurity-india-chennai-defenderrabbit-2025....
defencerabbit Team
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
[발표본] 너의 과제는 클라우드에 있어_KTDS_김동현_20250524.pdf
ssuserf8b8bd1
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
Teaching material agriculture food technology
LiaRayya
 
Machine learning based COVID-19 study performance prediction
IAESIJAI
 
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
breach-and-attack-simulation-cybersecurity-india-chennai-defenderrabbit-2025....
defencerabbit Team
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 

OCF.tw's talk about "Introduction to spark"

  • 1. Introduction to Spark Wisely Chen (aka thegiive) Sr. Engineer at Yahoo
  • 2. Agenda • What is Spark? ( Easy ) • Spark Concept ( Middle ) • Break : 10min • Spark EcoSystem ( Easy ) • Spark Future ( Middle ) • Q&A
  • 3. Who am I? • Wisely Chen ( thegiive@gmail.com ) • Sr. Engineer in Yahoo![Taiwan] data team • Loves to promote open source tech • Hadoop Summit 2013 San Jose • Jenkins Conf 2013 Palo Alto • Spark Summit 2014 San Francisco • Coscup 2006, 2012, 2013 , OSDC 2007, Webconf 2013, Coscup 2012, PHPConf 2012 , RubyConf 2012
  • 4. Taiwan Data Team Data! Highway BI! Report Serving! API Data! Mart ETL / Forecast Machine! Learning
  • 8. Opinion from Cloudera • The leading candidate for “successor to MapReduce” today is Apache Spark • No vendor — no new project — is likely to catch up. Chasing Spark would be a waste of time, and would delay availability of real-time analytic and processing services for no good reason. ! • From http://0rz.tw/y3OfM
  • 9. What is Spark • From UC Berkeley AMP Lab • Most activity Big data open source project since Hadoop
  • 13. YARN HDFS MapReduce Hadoop 2.0 Storm HBase Others
  • 14. Hadoop Architecture Hive MapReduce YARN HDFS SQL Computing Engine Resource Management Storage
  • 15. Hadoop vs Spark Hive Shark/SparkSQL YARN HDFS MapReduce Spark
  • 16. Spark vs Hadoop • Spark run on Yarn, Mesos or Standalone mode • Spark’s main concept is based on MapReduce • Spark can read from • HDFS: data locality • HBase • Cassandra
  • 17. More than MapReduce Shark: Hive GraphX: Pregel MLib: Mahout Spark Core : MapReduce HDFS Streaming: Storm Resource Management System(Yarn, Mesos)
  • 20. Logistic regression 3 110 82.5 55 27.5 33 106 180 135 90 45 171 3X~25X than MapReduce framework ! From Matei’s paper: http://0rz.tw/VVqgP Running Time(S) 80 60 40 20 0 76 MR Spark KMeans 0 MR Spark PageRank 0 23 MR Spark
  • 21. What is Spark • Apache Spark™ is a very fast and general engine for large-scale data processing
  • 22. Language Support • Python • Java • Scala
  • 23. Python Word Count • file = spark.textFile("hdfs://...") • counts = file.flatMap(lambda line: line.split(" ")) • .map(lambda word: (word, 1)) • .reduceByKey(lambda a, b: a + b) • counts.saveAsTextFile("hdfs://...") Access data via Spark API Process via Python
  • 24. What is Spark • Apache Spark™ is a very fast and general engine for large-scale data processing
  • 25. Why is Spark so fast?
  • 26. Most machine learning algorithms need iterative computing
  • 27. a 1.0 1.0 1.0 1.0 PageRank b b 1st Iter 2nd Iter 3rd Iter b d c Rank Tmp Result Rank Tmp Result a 1.85 1.0 0.58 d c 0.58 a 1.31 1.72 0.39 d c 0.58
  • 28. HDFS is 100x slower than memory Input (HDFS) Iter 1 Tmp (HDFS) Iter 2 Tmp (HDFS) Iter N Input (HDFS) Iter 1 Tmp (Mem) Iter 2 Tmp (Mem) Iter N MapReduce Spark
  • 29. 3rd iteration(mem)! take 7.7 sec 2nd iteration(mem)! take 7.4 sec First iteration(HDFS)! take 200 sec Page Rank algorithm in 1 billion record url
  • 34. RDD • Resilient Distributed Dataset • Collections of objects spread across a cluster, stored in RAM or on Disk • Built through parallel transformations
  • 36. RDD val b = a.filer( line=>line.contain(“Spark”) ) RDD a RDD b val a =sc.textFile(“hdfs://....”) Value c val c = b.count() Transformation Action
  • 37. Log mining val a = sc.textfile(“hdfs://aaa.com/a.txt”)! val err = a.filter( t=> t.contains(“ERROR”) )! .filter( t=>t.contains(“2014”)! ! err.cache()! err.count()! ! val m = err.filter( t=> t.contains(“MYSQL”) )! ! ! .count()! val a = err.filter( t=> t.contains(“APACHE”) )! ! ! .count() Driver Worker! ! ! ! Worker! ! ! Tas! k Worker! ! ! ! Task Task
  • 38. Log mining val a = sc.textfile(“hdfs://aaa.com/a.txt”)! val err = a.filter( t=> t.contains(“ERROR”) )! .filter( t=>t.contains(“2014”)! ! err.cache()! err.count()! ! val m = err.filter( t=> t.contains(“MYSQL”) )! ! ! .count()! val a = err.filter( t=> t.contains(“APACHE”) )! ! ! .count() Driver Worker! ! ! ! RDD a Bloc! k1 Worker! ! ! ! RDD a Bloc! k3 Worker! ! ! ! RDD a Bloc! k2
  • 39. Log mining val a = sc.textfile(“hdfs://aaa.com/a.txt”)! val err = a.filter( t=> t.contains(“ERROR”) )! .filter( t=>t.contains(“2014”)! ! err.cache()! err.count()! ! val m = err.filter( t=> t.contains(“MYSQL”) )! ! ! .count()! val a = err.filter( t=> t.contains(“APACHE”) )! ! ! .count() Driver Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Block3 Worker! ! ! ! ! RDD err Block1 Block2
  • 40. Log mining val a = sc.textfile(“hdfs://aaa.com/a.txt”)! val err = a.filter( t=> t.contains(“ERROR”) )! .filter( t=>t.contains(“2014”)! ! err.cache()! err.count()! ! val m = err.filter( t=> t.contains(“MYSQL”) )! ! ! .count()! val a = err.filter( t=> t.contains(“APACHE”) )! ! ! .count() Driver Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Block3 Worker! ! ! ! ! RDD err Block1 Block2
  • 41. Log mining val a = sc.textfile(“hdfs://aaa.com/a.txt”)! val err = a.filter( t=> t.contains(“ERROR”) )! .filter( t=>t.contains(“2014”)! ! err.cache()! err.count()! ! val m = err.filter( t=> t.contains(“MYSQL”) )! ! ! .count()! val a = err.filter( t=> t.contains(“APACHE”) )! ! ! .count() Driver Worker! ! ! ! ! RDD err Worker! ! ! ! ! RDD err Cache3 Worker! ! ! ! ! RDD err Cache1 Cache2
  • 42. Log mining val a = sc.textfile(“hdfs://aaa.com/a.txt”)! val err = a.filter( t=> t.contains(“ERROR”) )! .filter( t=>t.contains(“2014”)! ! err.cache()! err.count()! ! val m = err.filter( t=> t.contains(“MYSQL”) )! ! ! .count()! val a = err.filter( t=> t.contains(“APACHE”) )! ! ! .count() Driver Worker! ! ! ! ! RDD m Worker! ! ! ! ! RDD m Cache3 Worker! ! ! ! ! RDD m Cache1 Cache2
  • 43. Log mining val a = sc.textfile(“hdfs://aaa.com/a.txt”)! val err = a.filter( t=> t.contains(“ERROR”) )! .filter( t=>t.contains(“2014”)! ! err.cache()! err.count()! ! val m = err.filter( t=> t.contains(“MYSQL”) )! ! ! .count()! val a = err.filter( t=> t.contains(“APACHE”) )! ! ! .count() Driver Worker! ! ! ! ! RDD a Worker! ! ! ! ! RDD a Cache3 Worker! ! ! ! ! RDD a Cache1 Cache2
  • 44. RDD Cache with cache! take 7 sec 1st iteration(no cache)! take same time
  • 45. RDD Cache • Data locality • Cache After cache, take only 265ms A big shuffle! take 20min self join 5 billion record data
  • 46. Scala Word Count • val file = spark.textFile("hdfs://...") • val counts = file.flatMap(line => line.split(" ")) • .map(word => (word, 1)) • .reduceByKey(_ + _) • counts.saveAsTextFile("hdfs://...")
  • 47. Step by Step • file.flatMap(line => line.split(" “)) => (aaa,bb,cc) • .map(word => (word, 1)) => ((aaa,1),(bb,1)..) • .reduceByKey(_ + _) => ((aaa,123),(bb,23)…)
  • 48. Java Wordcount • JavaRDD<String> file = spark.textFile("hdfs://..."); • JavaRDD<String> words = file.flatMap(new FlatMapFunction<String, String>() • public Iterable<String> call(String s) { return Arrays.asList(s.split(" ")); } • }); • JavaPairRDD<String, Integer> pairs = words.map(new PairFunction<String, String, Integer>() • public Tuple2<String, Integer> call(String s) { return new Tuple2<String, Integer>(s, 1); } • }); • JavaPairRDD<String, Integer> counts = pairs.reduceByKey(new Function2<Integer, Integer>() • public Integer call(Integer a, Integer b) { return a + b; } • }); • counts.saveAsTextFile("hdfs://...");
  • 49. Java vs Scala • Scala : file.flatMap(line => line.split(" ")) • Java version : • JavaRDD<String> words = file.flatMap(new FlatMapFunction<String, String>() • public Iterable<String> call(String s) { • return Arrays.asList(s.split(" ")); } • });
  • 50. Python • file = spark.textFile("hdfs://...") • counts = file.flatMap(lambda line: line.split(" ")) • .map(lambda word: (word, 1)) • .reduceByKey(lambda a, b: a + b) • counts.saveAsTextFile("hdfs://...")
  • 51. Highly Recommend • Scala : Latest API feature, Stable • Python • very familiar language • Native Lib: NumPy, SciPy
  • 52. How to use it? • 1. go to https://spark.apache.org/ • 2. Download and unzip it • 3. ./sbin/start-all.sh or ./bin/spark-shell
  • 53. DEMO
  • 58. Spark ECOSystem SparkSQL: Hive GraphX: Pregel MLib: Mahout Spark Core : MapReduce HDFS Streaming: Storm Resource Management System(Yarn, Mesos)
  • 60. Detail Streaming BI ETL Spark SparkSQL MLlib Hive HDFS Cassandra RDBMS
  • 63. Write once, Run use case
  • 64. BI (SparkSQL) Streaming (SparkStreaming) Machine Learning (MLlib) Spark
  • 66. Data Analyst Data Engineer Data Scientist
  • 67. Bridge people together • Scala : Engineer • Java : Engineer • Python : Data Scientist , Engineer • R : Data Scientist , Data Analyst • SQL : Data Analyst
  • 68. Yahoo EC team Data Platform! ! ! ! ! ! ! ! ! ! Filtered Data! (HDFS) Data Mart! (Oracle) ML Model! (Spark) BI Report! (MSTR) Traffic! Data Transaction! Data Shark
  • 70. Data Analyst 350 TB data • Select tweet from tweets_data where similarity(tweet , “FIFA” ) > 0.01 Machine ! Learning • = ! • http://youtu.be/lO7LhVZrNwA?list=PL-x35fyliRwiST9gF7Z8Nu3LgJDFRuwfr https://www.youtube.com/watch?v=lO7LhVZrNwA&list=PL-x35fyliRwiST9gF7Z8Nu3LgJDFRuwfr#t=2900
  • 71. Data Scientist http://goo.gl/q5CAx8 http://research.janelia.org/zebrafish/
  • 72. SQL (Data Analyst) Cloud Computing (Data Engineer) Machine Learning (Data Scientist) Spark
  • 74. BI (SparkSQL) Streaming (SparkStreaming) Machine Learning (MLlib) Spark
  • 75. Instant BI Report http://youtu.be/dJQ5lV5Tldw?t=30m30s
  • 76. BI (SparkSQL) Streaming (SparkStreaming) Machine Learning (MLlib) Spark
  • 77. Background Knowledge • Tweet real time data store into SQL database • Spark MLLib use Wikipedia data to train a TF-IDF model • SparkSQL select tweet and filter by TF-IDF model • Generate live BI report
  • 78. Code • val wiki = sql(“select text from wiki”) • val model = new TFIDF() • model.train(wiki) • registerFunction(“similarity” , model.similarity _ ) • select tweet from tweet where similarity(tweet, “$search” > 0.01 )
  • 80. Q & A