Bigdata and Hadoop
- 2. What is the Need of Big data Technology when we have robust, high-performing, relational database management system ?
- 3. Data Stored in structured format like PK, Rows, Columns, Tuples and FK . It was for just Transactional data analysis. Later using Data warehouse for offline data. (Analysis done within Enterprise) Massive use of Internet and Social Networking(FB, Linkdin) Data become less structured. Data is stored on central server.
- 5. âBig Dataâ is similar to âsmall dataâ, but bigger âŚbut having data bigger it requires different approaches: âş Techniques, tools and architecture âŚwith an aim to solve new problems âş âŚor old problems in a better way
- 7. HADOOP
- 8. ď§ Open-source data storage and processing API ď§ Massively scalable, automatically parallelizable ďŻ Based on work from Google ďŻGFS + MapReduce + BigTable ďŻ Current Distributions based on Open Source and Vendor Work ďŻApache Hadoop ďŻCloudera â CH4 w/ Impala ďŻHortonworks ďŻMapR ďŻAWS ďŻWindows Azure HDInsight
- 10. HDFS is a file system written in Java Sits on top of a native file system Provides redundant storage for massive amounts of data Use cheap, unreliable computers
- 11. Data is split into blocks and stored on multiple nodes in the cluster âş Each block is usually 64 MB or 128 MB (conf) Each block is replicated multiple times (conf) âş Replicas stored on different data nodes Large files, 100 MB+
- 12. Master Nodes Slave Nodes
- 13. NameNode ⺠only 1 per cluster ⺠metadata server and database ⺠SecondaryNameNode helps with some housekeeping ⢠JobTracker ⢠only 1 per cluster ⢠job scheduler
- 14. DataNodes ⺠1-4000 per cluster ⺠block data storage ⢠TaskTrackers ⢠1-4000 per cluster ⢠task execution
- 15. A single NameNode stores all metadata Filenames, locations on DataNodes of each block, owner, group, etc. All information maintained in RAM for fast lookup File system metadata size is limited to the amount of available RAM on the NameNode
- 16. DataNodes store file contents Stored as opaque âblocksâ on the underlying filesystem Different blocks of the same file will be stored on different DataNodes Same block is stored on three (or more) DataNodes for redundancy
- 17. DataNodes send heartbeats to NameNode âş After a period without any heartbeats, a DataNode is assumed to be lost âş NameNode determines which blocks were on the lost node âş NameNode finds other DataNodes with copies of these blocks âş These DataNodes are instructed to copy the blocks to other nodes âş Replication is actively maintained
- 18. The Secondary NameNode is not a failover NameNode Does memory-intensive administrative functions for the NameNode Should run on a separate machine
- 19. datanode daemon Linux file system ⌠tasktracker slave node datanode daemon Linux file system ⌠tasktracker slave node datanode daemon Linux file system ⌠tasktracker slave node namenode namenode daemon job submission node jobtracker
- 20. MapReduce
- 21. MapReduce JobTracker MapReduce job submitted by client computer Master node TaskTracker Slave node Task instance TaskTracker Slave node Task instance TaskTracker Slave node Task instance In our case: circe.rc.usf.edu
- 22. Preparing for MapReduce Loading Files 64 MB 128 MB File System Native file system HDFS Cloud Output Immutable You Define Input, Map, Reduce, Output Use Java or other programming language Work with key- value pairs
- 23. Input: a set of key/value pairs User supplies two functions: âş map(k,v) ď list(k1,v1) âş reduce(k1, list(v1)) ď v2 (k1,v1) is an intermediate key/value pair Output is the set of (k1,v2) pairs
- 24. InputFormat Map function Partitioner Sorting & Merging Combiner Shuffling Merging Reduce function OutputFormat
- 25. ⢠Task Tracker ⢠Data Node ⢠Task Tracker ⢠Data Node ⢠Task Tracker ⢠Data Node ⢠Name Node ⢠Job Tracker Master Node Slave Node 1 Slave Node 2 Slave Node 3
- 27. InputFormat: âş TextInputFormat âş KeyValueTextInputFormat âş SequenceFileInputFormat OutputFormat: âş TextOutputFormat âş SequenceFileOutputFormat
- 28. 2 8
- 29. Probably the most complex aspect of MapReduce! Map side âş Map outputs are buffered in memory in a circular buffer âş When buffer reaches threshold, contents are âspilledâ to disk âş Spills merged in a single, partitioned file (sorted within each partition): combiner runs here Reduce side âş First, map outputs are copied over to reducer machine âş âSortâ is a multi-pass merge of map outputs (happens in memory and on disk): combiner runs here âş Final merge pass goes directly into reducer
- 30. Mapper Reducer other mappers other reducers circular buffer (in memory) spills (on disk) merged spills (on disk) intermediate files (on disk) Combiner Combiner
- 31. Writable Defines a de/serialization protocol. Every data type in Hadoop is a Writable. WritableComprable Defines a sort order. All keys must be of this type (but not values). IntWritable LongWritable Text ⌠Concrete classes for different data types. SequenceFiles Binary encoded of a sequence of key/value pairs
- 32. Map function Reduce function Run this program as a MapReduce job
- 33. Hadoop Cluster You 1. Load data into HDFS 2. Develop code locally 3. Submit MapReduce job 3a. Go back to Step 2 4. Retrieve data from HDFS
- 34. Applications for Big Data Analytics Homeland Security FinanceSmarter Healthcare Multi-channel sales Telecom Manufacturing Traffic Control Trading Analytics Fraud and Risk Log Analysis Search Quality Retail: Churn, NBO
- 35. CASE STUDY : 1 Environment Change Prediction to Assist Formers Using Hadoop