Big Data Hadoop Technology
- 1. Big Data Hadoop Presented by:- Rahul Sharma B-Tech(Cloud Technology) 2nd year Poornima University (I.Nurture)
- 2. What is Hadoop? Hadoop is an open-source software framework for storing data and running applications on clusters of commodity hardware. It provides massive storage for any kind of data, enormous processing power and the ability to handle virtually limitless tasks or jobs.
- 3. What is the use of Hadoop technology? Hadoop is an open source, Java-based programming framework that supports the processing and storage of extremely large data sets in a distributed computing environment. It is part of the Apache project sponsored by the Apache Software Foundation
- 4. Why is Hadoop important? Ability to store and process huge amounts of any kind of data, quickly- With data volumes and varieties constantly increasing, especially from social media and the Internet of Things (IoT), that's a key consideration. Computing power- Hadoop's distributed computing model processes big data fast. The more computing nodes you use, the more processing power you have. Flexibility- Unlike traditional relational databases, you don’t have to preprocess data before storing it. You can store as much data as you want and decide how to use it later. That includes unstructured data like text, images and videos. Low cost- The open-source framework is free and uses commodity hardware to store large quantities of data. Scalability- You can easily grow your system to handle more data simply by adding nodes. Little administration is required.
- 5. What are the challenges of using Hadoop? Full-fledged data management and governance- Hadoop does not have easy-to-use, full-feature tools for data management, data cleansing, governance and metadata. Especially lacking are tools for data quality and standardization. Data security- Another challenge centers around the fragmented data security issues, though new tools and technologies are surfacing. The Kerberos authentication protocol is a great step toward making Hadoop environments secure.
- 6. How Is Hadoop Being Used? Low-cost storage and data archive- The modest cost of commodity hardware makes Hadoop useful for storing and combining data such as transactional, social media, sensor, machine, scientific, click streams, etc. The low-cost storage lets you keep information that is not deemed currently critical but that you might want to analyze later. IoT and Hadoop- Things in the IoT need to know what to communicate and when to act. At the core of the IoT is a streaming, always on torrent of data. Hadoop is often used as the data store for millions or billions of transactions. You can then continuously improve these instructions, because Hadoop is constantly being updated with new data that doesn’t match previously defined patterns.
- 7. How Is Hadoop Being Used? Complement your data warehouse- We're now seeing Hadoop beginning to sit beside data warehouse environments, as well as certain data sets being offloaded from the data warehouse into Hadoop or new types of data going directly to Hadoop. The end goal for every organization is to have a right platform for storing and processing data of different schema, formats, etc. to support different use cases that can be integrated at different levels.
- 8. How It Works and a Hadoop Glossary. "Currently, four core modules are included in the basic framework from the Apache Foundation:" Hadoop Common – the libraries and utilities used by other Hadoop modules. Hadoop Distributed File System (HDFS) – the Java-based scalable system that stores data across multiple machines without prior organization. YARN – (Yet Another Resource Negotiator) provides resource management for the processes running on Hadoop. MapReduce – a parallel processing software framework. It is comprised of two steps. Map step is a master node that takes inputs and partitions them into smaller subproblems and then distributes them to worker nodes. After the map step has taken place, the master node takes the answers to all of the subproblems and combines them to produce output.
- 10. Thank You. !