Big data in Azure

Editor's Notes

• #6: HDP is 100% compatible with Apache Hadoom Open Enterprise Hadoop Data Platform – Enterprise ready HDInsight – Available to Azure subscribers. Runs on Azure clusters to run HDP and integrates with Azure storage HDP for Windows – OnPrem solution for running Hadoop in Windows Server. Can be physical or virtual machines Microsoft Analytics Platform – Massively Parallel Processing (MPP) in Microsoft Parallel data warehouse (PDW) with Hadoop.
• #9: Cluster - The cluster is managed by a server called the name node that has knowledge of all the cluster servers and the parts of the data files stored on each one. To store incoming data, the name node server directs the client to the appropriate data node server. The name node also manages replication of data files across all the other cluster members that communicate with each other to replicate the data. DataStore – Key/Value store, Document Store (XML or JSON), Binary store, Column stores, Graph store
• #10: Cassandra™: A scalable multi-master database with no single points of failure.  Chukwa™: A data collection system for managing large distributed systems.  HBase™: A scalable, distributed database that supports structured data storage for large tables.  HCatalog™: A table and storage management service for data created using Apache Hadoop.  Hive™: A data warehouse infrastructure that provides data summarization and ad hoc querying.  Mahout™: A Scalable machine learning and data mining library.  Pig™: A high-level data-flow language and execution framework for parallel computation.  ZooKeeper™: A high-performance coordination service for distributed applications.
• #11: When you store your data using HDFS, it's contained within the nodes of the cluster and it must be called through the HDFS API. When the cluster is decommissioned, the data is lost as well. Azure Storage provides several advantages: you can load the data using standard tools, retain the data when you decommission the cluster, the cost is less, and other processes in Azure. HBase is a NoSQL wide-column data store implemented as distributed system that provides data processing and storage over multiple nodes in a Hadoop cluster. It provides a random, real-time, read/write data store designed to host tables that can contain billions of rows and millions of columns.
• #12: Pig - Pig is another framework. Using the Pig Latin language, you can create MapReduce jobs more easily than by having to code the Java yourself—the language has simple statements for loading data, storing it in intermediate steps, and computing the data, and it's MapReduce aware Hive - When you want to work with the data on your cluster in a relational-friendly format. Hive allows you to create a data warehouse on top of HDFS or other file systems and uses a language called HiveQL, which has a lot in common with the Structured Query Language (SQL). Mahout is a machine learning library, which allows you to perform data mining queries that examine data files to extract specific types of information. For example, it supports recommendation mining (finding user’s preferences from their behavior), clustering (grouping documents with similar topic content), and classification (assigning new documents to a category based on existing categorization). Storm is a distributed real-time computation system for processing fast, large streams of data. It allows you to build trees and directed acyclic graphs (DAGs) that asynchronously process data items using a user-defined number of parallel tasks. It can be used for real-time analytics, online machine learning, continuous computation, distributed RPC, ETL, and more.
• #13: Interactive Data Ingestion – Small amount of data Automative Batch Upload to HDInsight – Using SSIS to gather disaparate data sources and push them to HDINsight Relational data - Sqoop – For loading relational data into HDInsight Web log files - Flume
• #15: Built on top of Azure’s hyperscale network and supports both single files that can be multiple petabytes in size,