5 reasons why spark is in demand!

www.edureka.co/r-for-analytics
www.edureka.co/apache-spark-scala-training
5 Reasons why Spark is in demand !

Slide 2Slide 2 www.edureka.co/apache-spark-scala-training
Agenda
At the end of this webinar you will be able to know about:
 Reason #1 : Low Latency
 Reason #2 : Streaming Support
 Reason #3 : Machine Learning and Graph
 Reason #4 : Data Frame API introduction
 Reason #5 : Spark integration with hadoop

Spark Architecure
Machine Learning
Library
Graph
programming
Spark interface
For RDBMS lovers
Utility for
continues
ingestion of data

Low Latency

Spark tries to keep things in-memory of its distributed workers, allowing for significantly faster/lower-latency
computations, whereas MapReduce keeps shuffling things in and out of disk.
Sparks Cuts Down Read/Write I/O To Disk
Spark is good for data that fits in memory and off memory

The previous world record was 72 minutes, set by Yahoo using a Hadoop MapReduce cluster of 2100 nodes
Using Spark on 206 EC2 nodes, spark completed the benchmark in 23 minutes.
Spark sorted the same data 3X faster using 10X fewer machines
All the sorting took place on disk (HDFS), without using Spark’s in-memory cache.
How Fast A System Can Sort 100 TB Of Data

2014, 4.27 TB/min
100 TB in 1,406 seconds
207 Amazon EC2 i2.8xlarge nodes x
(32 vCores - 2.5Ghz Intel Xeon E5-2670
v2, 244GB memory, 8x800 GB SSD)
Reynold Xin, Parviz Deyhim, Xiangrui
Meng,
Ali Ghodsi, Matei Zaharia
Courtesy : sortbenchmark.org/
Sparks Benchmark

Streaming Support

Used for processing the real-time streaming data.
It uses the DStream : a series of RDDs, to process the real-time data
support streaming analytics reasonably well.
The Spark Streaming API closely matches that of the Spark Core
Event processing

Machine Learning and graph
implementation with DAG

MLlib,a machine
learning library
classification regression clustering collaborative filtering and so on
Some of these algorithms also work with streaming data, such as linear regression using ordinary least squares or k-means clustering
Machine Learning

Cyclic Data Flows
• All jobs in spark comprise a series of operators and run on a set of data.
• All the operators in a job are used to construct a DAG (Directed Acyclic
Graph).
• The DAG is optimized by rearranging and combining operators where
possible.

Component for graphs and graph-parallel computation
Extends the Spark RDD by introducing a new Graph abstraction
Graph Algorithms
PageRank Connected Components Triangle Counting
GraphX

Support for Data Frames

As spark continues to grow, it wants to enable wider audiences beyond “big data” engineers to leverage the
power of distributed processing.
Inspired by data frames in r and python (pandas)
Dataframes API is designed to make big data processing on tabular data easier
Dataframe is a distributed collection of data organized into named columns.
Provides operations to filter, group, or compute aggregates, and can be used with spark sql.
Can be constructed from structured data files, existing rdds, tables in hive, or external databases.
DataFrame

Ability to scale from KBs to PBs
Support for a wide array of data formats and storage systems
State-of-the-art optimization and code generation through the spark SQL catalyst optimizer
Seamless integration with all big data tooling and infrastructure via spark
Apis for python, java, scala, and R (in development via sparkr)
DataFrame features

Spark can use HDFS
Spark can use YARN

Spark can leverage the resource negotiator of Hadoop framework i.e. YARN
Spark workloads can make use of Symphony scheduling policies and execute via YARN
Spark execution modes
Standalone Mesos HDFS
Spark Execution Platforms

Spark Features/Modules In Demand
Source: Typesafe

New Features In 2015
Data Frames 
• Similar API to data frames in R and Pandas
• Automatically optimised via Spark SQL
• Released in Spark 1.3
SparkR 
• Released in Spark 1.4
• Exposes DataFrames, RDD’s & ML library in R
Machine Learning Pipelines 
• High Level API
• Featurization
• Evaluation
• Model Tuning
External Data Sources 
• Platform API to plug Data-Sources into Spark
• Pushes logic into sources
Source: Databrix

Spark overview

5 reasons why spark is in demand!

More Related Content

What's hot (20)

Viewers also liked (8)

Similar to 5 reasons why spark is in demand! (20)

More from Edureka! (20)

Recently uploaded (20)

5 reasons why spark is in demand!

Editor's Notes