What's the Scoop on Hadoop? How It Works and How to WORK IT!
3 likes3,265 views
The document discusses Hadoop as a framework for distributed processing of large datasets, emphasizing its functionalities such as fault tolerance, batch processing, and data warehousing. It highlights MongoDB's integration as an operational store for analytics and details various use cases like behavioral analytics and fraud detection. Additionally, it explains MapReduce configuration and the role of Hadoop components like HDFS and YARN in managing resources within a cluster.
![11
Processing Sensor Data
{
"_id" : ObjectId("556172a53004b760dde8a488"),
”deviceId" : 556172530004,
"value" : 6205,
"timestamp" : ISODate(”2015-06-02T02:03:17.906Z"),
"loc" : [
-174.95596353219008,
40.654427078258834
]
} Average Sensor Value By
Device
Time Interval
Location Bucket](https://image.slidesharecdn.com/whatsthescoopwithhadoop-150605180150-lva1-app6892/85/What-s-the-Scoop-on-Hadoop-How-It-Works-and-How-to-WORK-IT-11-320.jpg)
![12
Processing Sensor Data
{
"_id" : ObjectId("556172a53004b760dde8a488"),
”deviceId" : 556172530004,
"value" : 6205,
"timestamp" : ISODate(”2015-06-02T02:03:17.906Z"),
"loc" : [
-174.95596353219008,
40.654427078258834
]
} Average Sensor Value By
Device
Time Interval
Location Bucket](https://image.slidesharecdn.com/whatsthescoopwithhadoop-150605180150-lva1-app6892/85/What-s-the-Scoop-on-Hadoop-How-It-Works-and-How-to-WORK-IT-12-320.jpg)
![13
Processing Sensor Data
{
"_id" : ObjectId("556172a53004b760dde8a488"),
”deviceId" : 556172530004,
"value" : 6205,
"timestamp" : ISODate(”2015-06-02T02:03:17.906Z"),
"loc" : [
-174.95596353219008,
40.654427078258834
]
} Average Sensor Value By
Device
Time Interval
Location Bucket](https://image.slidesharecdn.com/whatsthescoopwithhadoop-150605180150-lva1-app6892/85/What-s-the-Scoop-on-Hadoop-How-It-Works-and-How-to-WORK-IT-13-320.jpg)
![14
Processing Sensor Data
{
"_id" : ObjectId("556172a53004b760dde8a488"),
”deviceId" : 556172530004,
"value" : 6205,
"timestamp" : ISODate(”2015-06-02T02:03:17.906Z"),
"loc" : [
-174.95596353219008,
40.654427078258834
]
} Average Sensor Value By
Device
Time Interval
Location Bucket](https://image.slidesharecdn.com/whatsthescoopwithhadoop-150605180150-lva1-app6892/85/What-s-the-Scoop-on-Hadoop-How-It-Works-and-How-to-WORK-IT-14-320.jpg)
![15
Processing Sensor Data
{
"_id" : ObjectId("556172a53004b760dde8a488"),
"d_id" : ObjectId("556172a53004b760dde8a443"),
"v" : 6205,
"timestamp" : ISODate("3129-12-13T02:03:17.906Z"),
"loc" : [
-174.95596353219008,
40.654427078258834
]
}
LIVE CODE DEMO!!!
PSUEDO
^](https://image.slidesharecdn.com/whatsthescoopwithhadoop-150605180150-lva1-app6892/85/What-s-the-Scoop-on-Hadoop-How-It-Works-and-How-to-WORK-IT-15-320.jpg)
What's the Scoop on Hadoop? How It Works and How to WORK IT!
- 1. What’s the Scoop with Hadoop? How the connector works and how to work it { Name: ‘Bryan Reinero’, Title: ‘Developer Advocate’, Twitter: ‘@blimpyacht’, Email: ‘bryan@mongdb.com’ }
- 2. 2
- 3. 3 Hadoop A framework for distributed processing of large data sets • Terabyte and petabyte datasets • Data warehousing • Advanced analytics • Not a database • No indexes • Batch processing
- 5. 5 Data Management Hadoop Fault tolerance Batch processing Coarse-grained operations Unstructured Data MongoDB High availability Mutable data Fine-grained operations Flexible Schemas
- 6. 6 Data Management Hadoop Offline Processing Analytics Data Warehousing MongoDB Online Operations Application Operational
- 8. 8 MongoDB as an Operational Store Application Server
- 9. 9 Use Cases • Behavioral analytics • Segmentation • Fraud detection • Prediction • Pricing analytics • Sales analytics
- 10. What does it do?
- 11. 11 Processing Sensor Data { "_id" : ObjectId("556172a53004b760dde8a488"), ”deviceId" : 556172530004, "value" : 6205, "timestamp" : ISODate(”2015-06-02T02:03:17.906Z"), "loc" : [ -174.95596353219008, 40.654427078258834 ] } Average Sensor Value By Device Time Interval Location Bucket
- 12. 12 Processing Sensor Data { "_id" : ObjectId("556172a53004b760dde8a488"), ”deviceId" : 556172530004, "value" : 6205, "timestamp" : ISODate(”2015-06-02T02:03:17.906Z"), "loc" : [ -174.95596353219008, 40.654427078258834 ] } Average Sensor Value By Device Time Interval Location Bucket
- 13. 13 Processing Sensor Data { "_id" : ObjectId("556172a53004b760dde8a488"), ”deviceId" : 556172530004, "value" : 6205, "timestamp" : ISODate(”2015-06-02T02:03:17.906Z"), "loc" : [ -174.95596353219008, 40.654427078258834 ] } Average Sensor Value By Device Time Interval Location Bucket
- 14. 14 Processing Sensor Data { "_id" : ObjectId("556172a53004b760dde8a488"), ”deviceId" : 556172530004, "value" : 6205, "timestamp" : ISODate(”2015-06-02T02:03:17.906Z"), "loc" : [ -174.95596353219008, 40.654427078258834 ] } Average Sensor Value By Device Time Interval Location Bucket
- 15. 15 Processing Sensor Data { "_id" : ObjectId("556172a53004b760dde8a488"), "d_id" : ObjectId("556172a53004b760dde8a443"), "v" : 6205, "timestamp" : ISODate("3129-12-13T02:03:17.906Z"), "loc" : [ -174.95596353219008, 40.654427078258834 ] } LIVE CODE DEMO!!! PSUEDO ^
- 16. 16 MapReduce map() { emit( { key: ObjectId(…), value: 6205 } ); emit( { key: Chelsea, value: 6205 } ); emit( { key: m06_d01_h02, value: value } ); }
- 17. 17 MapReduce {key: ObjectId(…), value: 6205 } map() { emit( { key: ObjectId(…), value: 6205 } ); emit( { key: Chelsea, value: 6205 } ); emit( { key: m06_d01_h02, value: value } ); }
- 18. 18 MapReduce map() { emit( { key: ObjectId(…), value: 6205 } ); emit( { key: Chelsea, value: 6205 } ); emit( { key: m06_d01_h02, value: value } ); } {key: Chelsea, value: 6205}
- 19. 19 MapReduce map() { emit( { key: ObjectId(…), value: 6205 } ); emit( { key: Chelsea, value: 6205 } ); emit( { key: m06_d01_h02, value: value } ); } { key: m06_d01_h02, value: 6205}
- 20. 20 MapReduce
- 21. 21 MapReduce key: Chelsea, value: 6025
- 22. 22 MapReduce key: Chelsea, value: 4904
- 23. 23 MapReduce key: Chelsea, value: 6338
- 28. 28 MapReduce function reduce ( key, values ) { var result = { count: 1, sum : 0 }; values.forEach( function( v ){ result.sum = v.value; result.count++; }); return result; }
- 29. 29 MapReduce function reduce ( key, values ) { var result = { count: 1, sum : 0 }; values.forEach( function( v ){ result.sum = v.value; result.count++; }); return result; }
- 31. 31 HDFS and YARN • Hadoop Distributed File System (HDFS) – Distributed file-system that stores data on commodity machines in a Hadoop cluster • Yet Another Resource Negotiator (YARN) – Resource management platform responsible for managing and scheduling compute resources in a Hadoop cluster
- 32. 39 Hadoop Distributed File System (HDFS) DATA NODE DATA NODE DATA NODE DATA NODE Client Read / Writes Replication NAME NODE Metadata Operations
- 33. 47 Yet Another Resource Negotiator COMPUTE NODE Client NODE MANAGER NODE MANAGER RESOURCE MANAGER APPLICATION MASTER CONTAINER CONTAINER CONTAINER
- 35. 49 What You’re Gonna Need A reducer class extends org.apache.hadoop.mapreduce.Reducer A mapper class extends org.apache.hadoop.mapreduce.Mapper
- 36. 50 MapReduce Configuration • MongoDB input – mongo.job.input.format = com.mongodb.hadoop.MongoInputFormat – mongo.input.uri = mongodb://mydb:27017/db1.collection1 • MongoDB output – mongo.job.output.format = com.mongodb.hadoop.MongoOutputFormat – mongo.output.uri = mongodb://mydb:27017/db1.collection2 • BSON input/output – mongo.job.input.format = com.hadoop.BSONFileInputFormat – mapred.input.dir = hdfs:///tmp/database.bson – mongo.job.output.format = com.hadoop.BSONFileOutputFormat – mapred.output.dir = hdfs:///tmp/output.bson
- 37. 51 Yet Another Resource Negotiator COMPUTE NODE Client NODE MANAGER NODE MANAGER RESOURCE MANAGER APPLICATION MASTER CONTAINER CONTAINER CONTAINER Bin/hadoop jar MyJob.jar MongoDB_Hadoop_Connector.jar
- 38. 52
- 40. 54 extends MongoSplitter class List<InputSplit> calculateSplits()
- 41. 55 Cluster MONGOS SHARD A SHARDB SHARD C SHARD D MONGOS Client
- 42. 56 • High-level platform for creating MapReduce • Pig Latin abstracts Java into easier-to-use notation • Executed as a series of MapReduce applications • Supports user-defined functions (UDFs) Pig
- 43. 57 samples = LOAD 'mongodb://127.0.0.1:27017/sensor.logs' USING com.mongodb.hadoop.pig.MongoLoader(’deviceId:int,value:double'); grouped = GROUP samples by deviceId; sample_stats = FOREACH grouped { mean = AVG(samples.value); GENERATE group as deviceId, mean as mean; } STORE sample_stats INTO 'mongodb://127.0.0.1:27017/sensor.stats' USING com.mongodb.hadoop.pig.MongoStorage;
- 44. 58 • Data warehouse infrastructure built on top of Hadoop • Provides data summarization, query, and analysis • HiveQL is a subset of SQL • Support for user-defined functions (UDFs)
- 45. 59 • Powerful built-in transformations and actions – map, reduceByKey, union, distinct, sample, intersection, and more – foreach, count, collect, take, and many more An engine for processing Hadoop data. Can perform MapReduce in addition to streaming, interactive queries, and machine learning.
- 47. Thanks! { name: ‘Bryan Reinero’, title: ‘Developer Advocate’, twitter: ‘@blimpyacht’, code: ‘github.com/breinero’ email: ‘bryan@mongdb.com’ }