Big Data Workshop

Big Data:
Challenges and Opportunities
moshe.kaplan@brightaqua.com
http://blogs.microsoft.co.il/blogs/vprnd
http://top-performance.blogspot.com

http://top-performance.blogspot.com © Moshe Kaplan2

http://top-performance.blogspot.com © Moshe Kaplan
The Consumer Revolution
3
http://topyaps.com/wp-content/uploads/2013/03/You-are-the-product.-You-feeling-something.jpg

At the fraction of the cost…
4

http://lifehacker.com/5697167/if-youre-not-paying-for-it-youre-the-product

Transportation
6

Moovit
7

The Medical Market Opportunities
8

MediSafe
9

Askem
11

Major Enablers:
Mobile, Cloud and IT Commoditization
12

The Prime Suspect
13

Assumptions…

Where did it Fail?
Get an Answer, Fast and Cheap

Where did it Fail?
I Just Want “Class Persistency Storage”
and Changing Schema on Demand

Where did it Fail?
Be Always Available, Even w/ an Old Answer

Where did it Fail?
Get Me Fast and Good Enough Answer

Where did it Fail?
Data is Too Big, and Storage is $$$
But CPU and Network are Even More
http://www.powerbyte.com/Isilon.html

Software Providers
20

It is all great, but…
I Need to Meet Compliance
http://www.vision7.com/app_system/lib/image/content/PCI_compliance.jpg

I Need a Vendor

I Need Reporting
http://www.novell.com/communities/node/5851/get-ready-sentinel-61

I Need Transactions
http://www.novell.com/communities/node/5851/get-ready-sentinel-61

We Need Training for the Data Analysts
db.article.aggregate(
{ $group : {
_id : "$author",
docsPerAuthor : { $sum : 1 },
viewsPerAuthor : { $sum : "$pageViews" }
}}
);
< SUM(pageViews)
< SUM(1) = N
< GROUP BY author

The VP R&D Open Seminar
SCALING TO
HIGH THROUGHPUT
Redis
26

Key Value Store (with benefits)
insert
get
multiget
remove
truncate
27
<Key, Value>
http://wiki.apache.org/cassandra/API

1 Minute Installation
http://mariuszprzydatek.com/2014/08/23/amazon-aws-installing-
redis-on-ebs/
28

Fast. Very Fast
29

SCALING COMPLEX DATA
MongoDB
30

When Should I Choose NoSQL?
Eventually Consistent
Document Store
Key Value
31
http://guyharrison.squarespace.com/blog/tag/nosql

What mongoDB is Made of?
32
http://www.10gen.com/products/mongodb

Why MongoDB?
What? Why?
JSON End to End
No Schema “No DBA”, Just Serialize
Write 10K Inserts/sec on virtual machine
Read Similar to MySQL
HA 10 min to setup a cluster
Sharding Out of the Box
LBS Great for that
No Schema None: no downtime to create new columns
Buzz Trend is with NoSQL
33

DESIGN FOR NOSQL
MongoDB
34

Database for Software Engineers
Class
Subclass
Document
Subdocument

Same Terminology
Database  Database
Table  Collection
Row  Document

A Blog Case Study in MySQL
http://www.slideshare.net/nateabele/building-apps-with-mongodb

as a SW Engineer would like it to be…
http://www.slideshare.net/nateabele/building-apps-with-mongodb

CREATE YOUR SEARCH AND
ANALYTICS ENGINE
Elasticsearch (ELK)
39

Search Engine ++
Logstash
Elasticsearch:
Lucene based
Distributed + Map Reduce
Sharded
REST
Schemaless
Kibana
40

Combined Search
41

Oracle River
JDBC Connectivity
SP Support
Dual Mode:
Internal/Pull
Feeder/External/Push
42

Synonyms
Bv./Boulevard
Declared in the index
definition
Or at the synonyms file
43
PUT /my_index
{
"settings": {
"analysis": {
"filter": {
"my_synonym_filter": {
"type": "synonym",
"synonyms": [
"british,english",
"queen,monarch"
]
}
},
"analyzer": {
"my_synonyms": {
"tokenizer": "standard",
"filter": [
"lowercase",
"my_synonym_filter"
]
}
}
}
}
}

General Architecture and Status
44
API Client
Data
Store
The River
Data
Retrieval
Query
Indexed
Data
Prefix
Query
jQuery
AutoComplete

SCALING THE MAP REDUCE
Spark
45

HDFS: Physical Architecture
46

HDFS: Logical Architecture
47

HDFS+YARN
48

Map Reduce Concept
49

Map Reduce Example
50

WORD COUNT IN JAVA
Spark
51

Map Reduce Coding Headers
package org.myorg;
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import
org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
52

WordCount Class: Mapper
public static class Map extends Mapper<LongWritable, Text,
Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context
context) throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
context.write(word, one);
}
}
}
53

WordCount Class: Reduce
public static class Reduce extends Reducer<Text, IntWritable,
Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values,
Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
context.write(key, new IntWritable(sum));
}
}
54

WordCount Class: Main
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = new Job(conf, "wordcount");
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
}
55
Map and Reduce
Keys
Values

WORD COUNT IN SCALA
Spark
56

WordCount
57
http://www.slideshare.net/rhatr/apache-spark-killer-or-savior-of-apache-hadoop

“Hadoop” in Memory
58
http://spark.apache.org/

Product Stack
59
http://bigonlinenews.com/hortonworks-certifies-spark-on-yarn-hadoop/
Name Service
ZooKeeper
Log Shipping
Flume
Bulk Data
Sqoop
ML
Mahoot
Workflow
Oozie
Graph
Giraph

Streaming Example
62

MLlib
import org.apache.spark.mllib.linalg.Vector
import org.apache.spark.mllib.stat.
{MultivariateStatisticalSummary, Statistics}
val observations: RDD[Vector] = ...
// an RDD of Vectors
// Compute column summary statistics.
val summary: MultivariateStatisticalSummary =
Statistics.colStats(observations)
println(summary.mean)
// a dense vector containing the mean value for each
column
println(summary.variance)
// column-wise variance
println(summary.numNonzeros)
// number of nonzeros in each column
63
https://spark.apache.org/docs/latest/mllib-guide.html

Graphix
64
https://spark.apache.org/docs/latest/graphx-programming-guide.html

SCALING TO ALWAYS ON
Cassandra
65

Product Architecture
http://horicky.blogspot.co.il/2010/10/bigtable-model-with-cassandra-and-hbase.html

Key Concepts
Fast Answer  Use the memory
Not Always Right  Multiple instances
Can Lose Data  Multiple instances
Autosync  Client timestamp
Bottom Line: Integrated Memcached + MySQL

THE 5 STEPS FOR YOUR
BUSINESS SOLUTION
What’s Next?
68

1: Create Clear Value to Your Customers
69
http://cdn.business2community.com/wp-content/uploads/2014/07/Value_Creation.jpg

2: Collect Customers Insights
70
http://www.officescope.com/blog/wp-content/uploads/2013/07/Puzzle-pieces.jpg

3: Fusion the Data
71
http://www.officescope.com/blog/wp-content/uploads/2013/07/Puzzle-pieces.jpg

4: Create Value for Your (Real?) Customers
72
http://metro.co.uk/2010/09/28/supermarkets-big-value-packs-deemed-misleading-527232/

5: Implement Using the Right Tools
73

Moshe Kaplan +972-54-2291978
moshe.kaplan@brightaqua.com
http://top-performance.blogspot.com

Big Data Workshop

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Big Data Workshop (20)

More from Moshe Kaplan (20)

Recently uploaded (20)

Big Data Workshop