Introduing spark
0 likes164 views
This document introduces Spark, including when it was created, what it is, and why it was developed. Spark was created in 2009 at the AMPLab at UC Berkeley. It is now a top-level Apache project that provides a fast and general engine for large-scale data processing. It has high-level APIs for Scala, Python, R and Java and can be used for SQL, streaming, machine learning and graph processing. The document discusses Spark's programming model and demos its use for applications like Monte Carlo simulation and financial analysis.
![Copyright © 2014 DataYes. All rights reserved
Spark ! When, What, Why ?
From official: Apache Spark™ is a fast and general engine for large-scale data processing.
Key Points:
● A framework
● Birth for large-scale data processing
● Generalize programming model for data processing [ more than MR ]
● Provides high-level APIs : Scala, Python, R, Java
● Arm to teeth : SQL, Streaming, Machine Learning, GraphX
● Compatible with previous ecology : hadoop, mesos, hdfs, cassandra, hbase, s3 …](https://image.slidesharecdn.com/introduingspark-160510041659/85/Introduing-spark-5-320.jpg)
![Copyright © 2014 DataYes. All rights reserved
Demo & Next
● Wrapper Spark for Uqer Use Cases
●
● Try Tungsten
●
● Dataframe & Datasets
●
● SQL & Mlib & Streaming
●
● 3-party package wrapper [sklearn, pandas, numpy ...etc]](https://image.slidesharecdn.com/introduingspark-160510041659/85/Introduing-spark-13-320.jpg)
Introduing spark
- 3. Copyright © 2014 DataYes. All rights reserved Agenda Spark ! When, What, Why ?1 Basic Concepts in Spark2 Programming Model in Spark3 Demo & Next4 5 Q & A
- 4. Copyright © 2014 DataYes. All rights reserved Spark ! When, What, Why ? Top-level in Apache 2009 : Spark birth in AMPLab@UCB 2010 : open source Into Apache incubator 2009~2010 2013 2014 New Stage : more than an open source project
- 5. Copyright © 2014 DataYes. All rights reserved Spark ! When, What, Why ? From official: Apache Spark™ is a fast and general engine for large-scale data processing. Key Points: ● A framework ● Birth for large-scale data processing ● Generalize programming model for data processing [ more than MR ] ● Provides high-level APIs : Scala, Python, R, Java ● Arm to teeth : SQL, Streaming, Machine Learning, GraphX ● Compatible with previous ecology : hadoop, mesos, hdfs, cassandra, hbase, s3 …
- 6. Copyright © 2014 DataYes. All rights reserved Spark ! When, What, Why ? ● General ● Fast in develop ○ REPL explore ○ RDD operations ○ Less code ● Fast in processing ● Compatible ● Packges and 3-party packages ● Memory, cheaper and cheaper ● Company who accepts Spark
- 7. Copyright © 2014 DataYes. All rights reserved Spark ! When, What, Why ?
- 8. Copyright © 2014 DataYes. All rights reserved Spark ! When, What, Why ? DDR4-3000 288-pin DIMM 4x4GB Price Trend
- 9. Copyright © 2014 DataYes. All rights reserved Basic Concepts in Spark
- 10. Copyright © 2014 DataYes. All rights reserved Basic Concepts in Spark ● Driver, Master, Worker, Executor ● Application ● SparkContext, i.e : sc ● RDD ● Transform & Action in RDD need more ? check : 『 Spark 』2. spark 基本概念解析
- 11. Copyright © 2014 DataYes. All rights reserved Programming Model in Spark
- 12. Copyright © 2014 DataYes. All rights reserved Programming Model in Spark Three basic steps to build a Spark Application ● load dataset ○ static dataset ○ dynamic dataset ● Processing ○ RDD operation ○ UDF ○ Cache ● Output Display ○ collect ○ store in database, file system ...
- 13. Copyright © 2014 DataYes. All rights reserved Demo & Next ● Wrapper Spark for Uqer Use Cases ● ● Try Tungsten ● ● Dataframe & Datasets ● ● SQL & Mlib & Streaming ● ● 3-party package wrapper [sklearn, pandas, numpy ...etc]
- 14. Copyright © 2014 DataYes. All rights reserved Demo & Next
- 15. Copyright © 2014 DataYes. All rights reserved Demo & Next ● Monte Carlo in Spark ● Spark in finance : index similarity calculating ● Spark in finance : distributed backtesting strategy
- 16. Copyright © 2014 DataYes. All rights reserved Demo, Demo, Demo Q & A
- 17. 谢 谢