introduction to big data frameworks

Machines Learning in Big Data (
MapReduce, Knime, Spark)
Présenté par: Sahmoudi Yahia
Targhi Amal
24/12/2015
1
Proposé par : Bouchra Frikh

Outlines
 Introduction
 Big Data
 Machine Learning
 Applications of ML Techniques to Data mining Tasks
 Why Machine Learning in Big Data?
 Big Data frameworks processing
24/12/2015
2

Introduction
• Every day, 2.5 quintillion bytes of data are created and 90% of the data
in the world today were produced within the past two years (IBM 2012).
• On October 4, 2012, the first presidential debate between President
Barack Obama and Governor Mitt Romney triggered more than 10
million tweets within two hours (Twitter Blog 2012).
• Another example is Flickr, a public picture sharing site, which received
1.8 million photos per day, on average, from February to March 2012
(Michel F. 2012).
24/12/2015
3

• “ The most fundamental challenge for the Big
Data applications is to explore the large volumes
of data and extract useful information or
knowledge for future actions “ (Rajaraman and
Ullman, 2011).
24/12/2015
4

• What is big data?
• What is machine learning?
• Why do machine learning in big data?
• What are they the big data processing frameworks
which integrate machine learning algorithm?
24/12/2015
5

Big Data
• Big data is a term that describes the large volume of data (
structed, semi- structed, unstructed), which can exploited to
get informations.
• Big data can be characterized by 3Vs:
• the extreme volume of data.
• the wide variety of types of data
• velocity at which the data must be must processed.
24/12/2015
6

P.S and we can also add: Variability and Complexity.
24/12/2015
7

Big Data Analytics
Big data analytics is the process of examining large data
sets containing a variety of data types.
“ It’s important to remember that the primary value from big data
comes not from the data in its raw form, but from the processing and
analysis of it and the insights, products, and services that emerge from
analysis. The sweeping changes in big data technologies and
management approaches need to be accompanied by similarly dramatic
shifts in how data supports decisions and product/service innovation.”
Thomas H. Davenport in Big Data in Big Companies
24/12/2015
8

Machine Learning
Design off implementation methods allowing a machine to evolve by
a systematic process and completing tasks which are difficult
They are difficult for a classic algorithm.
Machine Learning it’s when de data replace the algortihm
Example
Facebook's News Feed changes according to the user's personal interactions with
other users. If a user frequently tags a friend in photos, writes on his wall or likes his
links, the News Feed will show more of that friend's activity in the user's News Feed
due to presumed closeness.
24/12/2015
9

Applications of ML Techniques to
Data mining Tasks
• The amount of the data seems to increase rapidly every single day for the
majority domains related to information processing, and the need to find a
way to mine and get knowledge from databases is still crucial.
• Data Mining (DM) defines the automated extraction procedures of hidden
predictive information from databases . Among the tasks of DM are:
• Diagnosis
• Pattern Recognition:
• Prediction
• Classification
• Clustering
• Optimization:
• Control:
24/12/2015
10

The impact of ML techniques used
for DM tasks
Artificial Neural Networks for DM
Genetic Algorithms for DM
Inductive Logic Programming for DM
Rule Induction for DM
Decision Trees for DM
Instance-based Learning Algorithms for DM
24/12/2015
11

Exemple
• Amazon is an interesting example of application
of "Machine Learning" in e-commerce. For
example, suppose we perform the search for a
product on Amazon today. When we come back
another day on the site, it is able to offer us
products related to our specific needs
(prediction). And thanks to algorithms of
"Machine Learning" which provide our changing
needs from our previous visits to the site.
24/12/2015
12

Why Machine Learning in big Data?
• It delivers on the promise of extracting value from big and
disparate data sources with far less reliance on human
direction. It is data driven and runs at machine scale. It is well
suited to the complexity of dealing with disparate data sources
and the huge variety of variables and amounts of data
involved. And unlike traditional analysis, machine learning
thrives on growing datasets. The more data fed into a machine
learning system, the more it can learn and apply the results to
higher quality insights.
24/12/2015
13

Big Data framework s processing
• The Machine Learning is far from being a recent
phenomenon. What is new, however, is the number of
treatments platforms parallelized data for the
management of Big Data.
• in this Presentation we will take a look on some of the most used
algorithms as MapReduce, knime, spark.
24/12/2015
14

MapReduce
• MapReduce invented by google, it's the heart of
hadoop and it's is a processing technique and a
program model for distributed computing based on
java.
• MapReduce allows for massive scalability across
hundreds or thousands of servers in a Hadoop
cluster., allows to write applications to process huge
amounts of data, in parallel, on large clusters of
commodity hardware in a reliable manner.
24/12/2015
15

• The MapReduce algorithm refers two important tasks,Map
and Reduce:
• Map: Map takes a set of data and converts it into another set
of data, where individual elements are broken down into
tuples (key/value pairs).
• Reduce: takes the output from a map as an input and
combines those data tuples into a smaller set of tuples. As the
sequence of the name MapReduce implies, the reduce task is
always performed after the map job.
24/12/2015
16

Algorithm
• Map stage : The map or mapper’s job is to process the input data. Generally
the input data is in the form of file or directory and is stored in the Hadoop
file system (HDFS). The input file is passed to the mapper function line by
line. The mapper processes the data and creates several small chunks of
data.
• Reduce stage : This stage is the combination of the Shuffle stage and
the Reduce stage. The Reducer’s job is to process the data that comes from
the mapper. After processing, it produces a new set of output, which will be
stored in the HDFS.
24/12/2015
17

Exemple
• The Word Count program used in this section to
explain how a map will also be used to explain
the operation of a reducer.
• the object of this program is: counting the
number of occurrences of the different words
which make up a book.
24/12/2015
19

Book Content
book's content
24/12/2015
20

Step 1 : Map
• Line 1: the mapper reads in input a record that comes in the form of
a pair <key, value> with:
• the String value type (a line of the file).
• LongWritable the key type
• Line 2 : for each word in the current line.
• Line 3 : we write on a output file the couple <w,1> , corresponding
to an occurrence of the word in the variable w.
24/12/2015
21

the output file of the mapper
24/12/2015
22

Step 2 ( between map and Reduce:
shuffle) and sort
24/12/2015
23

Step3 : Reduce
Line 1: the reducer input a recording as a pair <key, value> with:
Text the key type (one word).
value is a list of values of intWritable type.
Line 2: the reducer resets the counter when the wordcount word is changed.
Line 3: v for each value in the list is added to inValue2 v Word-Count (in this case v is
always 1).
Line4: When we change the word, we write in a output file: couple <inKey2, wordCount>,
wordCount is the number of occurrences of the word.
24/12/2015
24

Use case in real life
• http://hadoopilluminated.com/hadoop_book/Hadoop_Us
e_Cases.html#d1575e1290
24/12/2015
26

Spark
• Apache Spark is an open source cluster
computing framework originally developed in
the AMPLab at University of California,
• Spark extends the popular MapReduce model to
efficiently support more types of computations.
24/12/2015
27

A Unified Stack
• one of the largest advantages of tight integration is the ability to build
applications that seamlessly combine different processing models. For
example, in Spark you can write one application that uses machine learning
to classify data in real time as it is ingested from streaming sources.
Simultaneously, analysts can query the resulting data, also in real time, via
SQL (e.g., to join the data with unstructured logfiles).
• In addition, more sophisticated data engineers and data scientists can
access the same data via the Python shell for ad hoc analysis. Others might
access the data in standalone batch applications. All the while, the IT team
has to maintain only one system.
24/12/2015
28

Rdd
• the resilient distributed dataset (RDD). An RDD is simply
a distributed collection of elements. In Spark all work is
expressed as either creating new RDDs, transforming existing
RDDs, or calling operations on RDDs to compute a result. Under the
hood, Spark automatically distributes the data contained in RDDs
across your cluster and Parallelizes the operations you perform on
them.
• RDDs are the core concept in Spark.
• You can see a RDD as a table in a database.
24/12/2015
29

• Transformations: transformations do not return
a single value, they return a new RDD. Nothing is
evaluated when one uses a transformation function,
this function just takes and RDD and returns a new
RDD . The processing functions such as map, filter,
flatMap, groupByKey, reduceByKey,
aggregateByKey ...
• Actions: shares evaluate and return a new value.
When an action function is called on a RDD object,
all data processing requests are calculated and the
result is returned. The actions are reduce, collect,
count, first, take, countByKey and foreach ...
24/12/2015
30

Spark Core
Spark Core contains the basic functionality of Spark, including components for task
scheduling, memory management, fault recovery, interacting with storage systems,
and more. Spark Core is also home to the API that defines resilient distributed datasets
(RDDs), which are Spark’s main programming abstraction. RDDs represent a
collection of items distributed across many compute nodes that can be manipulated
in parallel. Spark Core provides many APIs for building and manipulating these
collections.
24/12/2015
32

Example in real Life
Another early Spark adopter is Conviva, one of the largest
streaming video companies on the Internet, with about 4
billion video feeds per month (second only to YouTube). As
you can imagine, such an operation requires pretty
sophisticated behind-the-scenes technology to ensure a
high quality of service. As it turns out, it’s using Spark to
help deliver that QoS by avoiding dreaded screen buffering.
24/12/2015
33

MapReuce Vs Spark
Spark MapReduce
Performance Spark processes data
in-memory
On the Disc
Compatibility Apache Spark can run
as standalone or on top
of Hadoop YARN or on
the cloud. ..
With hadoop
Data Processing Data Batch
Failure Tolerance + +
Security is still in its infancy Hadoop MapReduce
has more security
features and projects.n
it comes to security
24/12/2015
34

introduction to big data frameworks

More Related Content

What's hot (20)

Viewers also liked (18)

Similar to introduction to big data frameworks (20)

Recently uploaded (20)

introduction to big data frameworks

Editor's Notes