Big data, Hadoop and Hive
Download as PPTX, PDF1 like225 views
This document discusses big data and Hadoop. It begins by defining big data and explaining the volume, variety, and velocity of data being generated. It then provides an overview of Hadoop, including its origins, architecture, core components of HDFS and MapReduce, and Hadoop ecosystems like Hive. HDFS provides a distributed file system for large data storage, while MapReduce allows processing of large datasets in parallel across clusters. Hive allows querying and analysis of large datasets stored in Hadoop. Overall, the document serves as an introduction to big data concepts and the Hadoop framework for distributed storage and processing of large and diverse datasets.
Big data, Hadoop and Hive
- 1. Big Data – Srinath & Arjun
- 2. Big Data – Srinath & Arjun • The BIG-DATA • Hadoop • Hadoop Components • Hadoop Eco Systems 2 Agenda
- 3. Big Data – Srinath & Arjun The BIG-DATA
- 4. Big Data – Srinath & Arjun 4 The Context • Man on the moon with 32KB (1969); my laptop had 2GB RAM (2009) • Google collects 270PB data in a month (2007), 20000PB a day (2008) • 2010 census data is expected to be a huge gold mine of information • Data mining huge amounts of data collected in a wide range of domains from astronomy to healthcare has become essential for planning and performance.
- 5. Big Data – Srinath & Arjun • We are in a knowledge economy. – Data is an important asset to any organization – Discovery of knowledge; Enabling discovery; annotation of data • We are looking at newer – programming models, and – Supporting algorithms and data structures. The Context
- 6. Big Data – Srinath & Arjun • Big Data is New • Big Data is only about Massive Data Volume • Big data means Hadoop • Big data need a Data Warehouse • Big data means Unstructured Data • Big data is for Social Media and Data mining Analyses 6 The Myth about Big Data
- 7. Big Data – Srinath & Arjun It is all about better analytic on a broader spectrum of data, and therefore represents an opportunity to create even more differentiation among industries. 7 Big Data is…
- 8. Big Data – Srinath & Arjun Where Data is coming….? 12+ TBs of tweet data every day 25+ TBs of log data every day ?TBsof dataevery day 2+ billion people on the Web by end 2011 30 billion RFID tags today (1.3B in 2005) 4.6 billion camera phones world wide 100s of millions of GPS enabled devices sold annually 76 million smart meters in 2009… 200M by 2014
- 9. Big Data – Srinath & Arjun Facebook • 4.5 billion Facebook likes every day • 350 million photos uploaded on a daily basis • 250 billion photos stored by Facebook • 10 billion messages sent everyday • 1 trillion posts in Facebook’s graph search database • 500 TB of data processed daily • 100 PB of data stored in Facebok’s Hadoop disk cluster (1PB=1000TB=1000000 GB) Example of Big Data Generation
- 10. Big Data – Srinath & Arjun Flights • 1 Boeing plane engine generates 20TB of data for every hour of flying • How much data do all the flights in this world generate every year if there are 100000 two engine flights daily? Example of Big Data Generation
- 11. Big Data – Srinath & Arjun • Black Box Data • Social Media Data • Stock Exchange Data • Power Grid Data • Transport Data • Search Engine Data What comes under Big data?
- 12. Big Data – Srinath & Arjun • Capturing Data • Storage • Searching • Sharing • Transfer • Analysis • Presentation Big Data Challenges
- 13. Big Data – Srinath & Arjun Characteristics of Big Data Volume of Tweets create daily. 12+terabytes Variety of different types of data. 100’s Veracity decision makers trust their information. Only 1 in 3 trade events per second. 5+million Velocity
- 14. Big Data – Srinath & Arjun • Structured data : Relational Data • Semi Structured data : XML data • Unstructured Data : Word, PDF, Text, Media Logs Types of Data
- 15. Big Data – Srinath & Arjun The Data Explosion • 2.5 quintillion bytes of data created each year • 90 % of data in the world was created in the last two years
- 16. Big Data – Srinath & Arjun Hadoop
- 17. Big Data – Srinath & Arjun Hadoop • Open Source Software Framework • Inspired by Google’s Map – Reduce Programming Model (GFS) • Originally written for the Nutch search engine project • Written in java • Efficiently processes large volumes of Data • Breaks up Big data into multiple parts • Two key parts • HDFS • MapReduce
- 18. Big Data – Srinath & Arjun History of Hadoop
- 19. Big Data – Srinath & Arjun Hadoop Architecture
- 20. Big Data – Srinath & Arjun Hadoop Components
- 21. Big Data – Srinath & Arjun HDFS – Hadoop Distributed File System • It’s a file system designed for storing very large files running on cluster of commodity hardware • High fault tolerance, Distributed, Reliable, Scalable file system for Data Storage • Stores multiple copies of data on different nodes. (default 64MB) • Typically has a single namenode and no.of datanodes to form the HDFS cluster
- 22. Big Data – Srinath & Arjun HDFS Architecture • Two types of Nodes Master or Namenode Slave or Datanode
- 23. Big Data – Srinath & Arjun HDFS Architecture
- 24. Big Data – Srinath & Arjun Read a File
- 25. Big Data – Srinath & Arjun Write a File
- 26. Big Data – Srinath & Arjun Hadoop Cluster Modes • Standalone Mode • Pseudo-Distributed Mode • Fully-Distributed Mode
- 27. Big Data – Srinath & Arjun MapReduce Programming Model designed for processing large volumes of data in parallel by dividing the work into a set of independent tasks
- 28. Big Data – Srinath & Arjun Terminology • Job • Task • Task Attempt • NameNode • MasterNode • SlaveNode • Clusters • Commodity Hardware
- 29. Big Data – Srinath & Arjun Components • Master Nodes • Slave Nodes
- 30. Big Data – Srinath & Arjun Workflow
- 31. Big Data – Srinath & Arjun Example
- 32. Big Data – Srinath & Arjun Closer Look
- 33. Big Data – Srinath & Arjun Input Formats • Text Input Format • Sequential input format • Key value text input format
- 34. Big Data – Srinath & Arjun NoSQL • NoSQL mean “not only SQL” • This includes key value stores, document-oriented databases, graph databases, big datable structures, and caching data stores Eg. MongoDB, Cassandra
- 35. Big Data – Srinath & Arjun Hadoop ECO Systems
- 36. Big Data – Srinath & Arjun What is HIVE? • Data Warehousing Infrastructure • Data Summarization, ad-hoc querying and analysis of large volumes of data
- 37. Big Data – Srinath & Arjun HiveQL • HiveQL is the Hive query language. • Hive doesn’t support transactions.
- 38. Big Data – Srinath & Arjun Hive Application • Log Processing • Text Mining • Document indexing • Customer – facing Business intelligence (eg. Google Analytics) • Predictive modelling, hypothesis testing
- 39. Big Data – Srinath & Arjun Thank You….