UNIT-3 Data Visualization for the life used...
- 1. UNIT-3 Processing data visualization TOPICS ο Processing Big Data ο Integrating disparate data stores ο Mapping data to the programming framework ο Connecting and extracting data from storage ο Transforming data for processing ο subdividing data in preparation for Hadoop Map Reduce.
- 2. Processing Big Data Data processing means to process the data i.e. to convert its format. As we all know data is the very useful and when it is well presented, and it becomes informative and useful. Data processing system is also referred as information system.
- 4. Types of Data Processing β’ Manual Data Processing β’ Mechanical Data Processing β’ Electronic Data Processing β’ Batch Data Processing β’ Real-time Data Processing β’ Online Data Processing β’ Automatic Data Processing
- 6. Integrating disparate data stores Integrating disparate data stores involves combining and connecting data from different sources or storage systems to provide a unified and coherent view of the information. This process is crucial for organizations that have data scattered across various databases, file systems, or platforms
- 7. Here are the key aspects of integrating disparate data stores ο±Data Discovery ο±Data Mapping ο±Data Extraction ο±Data Transformation ο±Data Loading ο±Data Synchronization
- 8. HADOOP MAPREDUCE Mapping data to the programming framework
- 9. Map and Reduce Map: As the name suggests its main use is to map the input data in key-value pairs. The input to the map may be a key- value pair where the key can be the id of some kind of address and value is the actual value that it keeps. The Map() function will be executed in its memory repository on each of these input key-value pairs and generates the intermediate key-value pair which works as input for the Reducer or Reduce() function.
- 10. Reduce: The intermediate key-value pairs that work as input for Reducer are shuffled and sort and send to the Reduce() function. Reducer aggregate or group the data based on its key-value pair as per the reducer algorithm written by the developer.
- 12. The data mapping process in 5 steps ο±Identify all data fields that must be mapped ο±Standardize naming conventions across sources ο±Create data transformation rules and schema logic ο±Test your logic ο±Complete the migration, integration, or transformation
- 13. Connecting and extracting data from storage
- 14. Data EXTRACTION Software β’ Web Scraping Tools β’ OCR (Optical Character Recognition β’ PDF Extraction β’ Screen Scraping β’ Text Analytics
- 15. Data Transformation in Data Mining The goal of data transformation is to prepare the data for data mining so that it can be used to extract useful insights and knowledge.
- 16. Phases of MapReduce β How Hadoop MapReduce Works It covers all the phases of MapReduce job execution like Input Files, Input Format, Input Splits, Record Reader, Mapper, Combiner, Partitioner, Shuffling, and Sorting, Reducer, Record Writer, and Output Format in detail.
- 17. THANK YOU!!