Big data and tools
- 1. Big Data Presented by : SHIVAM SHUKLA
- 2. Contents ď´ What is Big data ? ď´ History ď´ Three Vâs ď´ Why Big Data important ? ď´ Technologies related to Big Data ď´Hadoop ď´Why Hadoop? ď´Hbase ď´Why Hbase? ď´Some features of Hbase
- 4. What is Big Data ? ď´ Big data is a term that describes the large volume of data : a) Structured b) Unstructured c) Semi-structured ď´ That inundates a business on a day-to-day basis. ď´ But itâs not the amount of data thatâs important. Itâs what organizations do with the data that matters.
- 5. History ď´ While the term âbig dataâ is relatively new, the act of gathering and storing large amounts of information for eventual analysis is ages old. ď´ The concept gained momentum in the early 2000s, when industry analyst Doug Laney articulated the now-mainstream definition of big data as the three Vs: ď´Volume ď´Velocity ď´Variety
- 6. Three Vâs : ď´ Volume Defines the huge amount of data that is produced each day by organizations in the world ď´ Velocity Refers to speed with which the data is generated , analyzed and reprocessed ď´ Variety refers to diversity of data and data sources
- 8. Additional Vâs With the time new Vâs of big data introduced ď´ Validity It refers to the guarantee of data quality or, alternatively, Veracity is the authenticity and credibility of the data. ď´ Value denotes the added value for companies. Many companies have recently established their own data platforms, filled their data pools and invested a lot of money in infrastructure. It is now a question of generating business value from their investments.
- 9. Why is Big Data important ? ď´ The importance of big data doesnât revolve around how much data you have, but what you do with it. ď´ You can take data from any source and analyze it to find answers that enable ď´Cost reduction ď´Time reduction ď´Smart decision making
- 10. Some Technologies related to Big data ď´ Hadoop framework ď´ Hbase ď´ Hive ď´ Scoop
- 11. Hadoop ď´ Hadoop is developed by Doug cutting and Michael j. cafarella. ď´ Hadoop is a apache open source frame work designed for ď´Managing the data ď´Processing the data ď´Analyzing the data ď´Storing the data ď´ Hadoop is written in java and not OLAP(online analytical processing). ď´ It is used for offline processing. ď´ Logo for Hadoop is a YELLOW ELEPHANT
- 12. Why Hadoop ? ď´ Fast : ď§ In HDFS the data distributed over the cluster and are mapped which helps in faster retrieval. ď´ Scalable : ď§ Hadoop cluster can be extended by just adding nodes in the cluster. ď´ Cost Effective : ď§ Hadoop is open source and uses commodity hardware to store data so it really cost effective as compared to traditional relational database management system. ď´ Resilient to failure : ď§ HDFS has the property with which it can replicate data over the network, so if one node is down or some other network failure happens, then Hadoop takes the other copy of data and use it.
- 13. HBase ď´ HBase is an open source framework provided by Apache. It is a sorted map data built on Hadoop. ď´ It is column oriented and horizontally scalable. ď´ It has set of tables which keep data in key value format. ď´ It is type of a database designed for mainly managing the unstructured data ď´ Logo for Apache HBase is a DOLPHIN
- 14. Why Hbase? ď´ RDBMS get exponentially slow as the data becomes large. ď´ Expects data to be highly structured, i.e. ability to fit in a well- defined schema. ď´ Any change in schema might require a downtime. ď´ For sparse datasets, too much of overhead of maintaining NULL values.
- 15. Some feature of Hbase ď´ Horizontally scalable: You can add any number of columns anytime. ď´ Often referred as a key value store or column family-oriented database, or storing versioned maps of maps. ď´ fundamentally, it's a platform for storing and retrieving data with random access. ď´ It doesn't care about datatypes(storing an integer in one row and a string in another for the same column). ď´ There is only one kind of data type which is byte array. ď´ It doesn't enforce relationships within your data. ď´ It is designed to run on a cluster of computers.
- 16. Hive ď´ Hive is a data warehouse infrastructure tool to process structured data in Hadoop. ď´ It runs SQL like queries called HQL (Hive query language) which gets internally converted to map reduce jobs. ď´ Initially Hive was developed by Facebook, later the Apache Software Foundation took it up and developed it further as an open source under the name Apache Hive. ď´ Hive supports Data definition Language(DDL), Data Manipulation Language(DML) and user defined functions. ď´ The logo for hive is a yellow and black BEE
- 17. Hive is not : ď´ A relational database ď´ designed for Online Transaction Processing (OLTP) ď´ A language for real-time queries and row-level updates ď´ Even with small amount of data ,time to return the response canât be compared to RDBMS.
- 18. Points to remember about hive ď´ Hive Query Language is similar to SQL and gets reduced to map reduce jobs in backend. ď´ Hive's default database is derby. ď´ It also called as a No Sql. ď´ It provides SQL type language for querying called HiveQL or HQL. ď´ It is designed for OLAP(Online analytics processing).
- 19. Sqoop ď´ Sqoop is a tool designed to transfer data between Hadoop and relational database servers. ď´ It is used to import data from relational databases such as MySQL, Oracle to Hadoop HDFS, and export from Hadoop file system to relational databases. ď´ It is provided by the Apache Software Foundation. ď´ Sqoop- âSQL to Hadoop and Hadoop to SQLâ
- 20. Working of sqoop
- 21. Difference Sqoop Import ď´ The import tool imports individual tables from RDBMS to HDFS. ď´ Each row in a table is treated as a record in HDFS. ď´ All records are stored as text data in text files or as binary data in Avro and Sequence files. Sqoop Export ď´ The export tool exports a set of files from HDFS back to an RDBMS. ď´ The files given as input to Sqoop contain records, which are called as rows in table. ď´ Those are read and parsed into a set of records and delimited with user-specified delimiter.