Big data & hadoop
- 1. Big Data & Hadoop
- 2. Agenda • 1 Introduction to big data • 2 Introduction to Hadoop • 3 HDFS • 4 Mapreduce • 5 YARN • 6 Hadoop ecosystem • 7 Hadoop installation • 8 MapReduce example • 9 Apache sqoop tutorial • 10 Apache flume tutorial • 11 Apace pig • 12 Apace hive • 13 Apace Hbase • 14 Hadoop project
- 3. Introduction to big data • Evolution of Technology • IOT • Social Media • Other Factors…
- 4. What is big Data? • Big Data is the term for collection of data sets so large and complex that it becomes difficult to process using on-hand database system tools or traditional data processing application
- 5. 5 v’s of Big Data • Volume of Data • Variety of Data structured,semi-structured,un-structured • Velocity • Value Mechanism to bring the correct meaning out of data • Veracity Uncertainty and inconsistencies in the data
- 6. Big Data Analytics • Big data analytics examines large and different types of data to uncover hidden patterns correlation and other insights. • Stages:- • Identify problem • Designing data requirement • Pre-processing data • Performing Analytics on data • Visualizing data
- 7. Types of Big Data Analytics • Descriptive Analysis • Predictive Analysis • Prescriptive Analytics • Diagnostic Analytics
- 9. Apace hadoop: Framework to process Big Data • Haddop is a framework that allows us to store and process large data sets in parlllel and distributed fashion.
- 11. HDFS • HDFS core components:- • Name Node • Data Node • Secondary Name node
- 12. Name Node and Data Node
- 13. Secondary NameNode & Checkpointing
- 14. HDFS Data Blocks
- 15. Fault Tolerance
- 16. HDFS Write Mechanism and Acknowledgement
- 19. MAP REDUCE
- 23. Mapper Code
- 24. Reduce Code
- 25. Driver Code
- 26. YARN
- 30. Cluster Modes