Introduction to Big Data
0 likes2,564 views
This document introduces big data concepts and Microsoft's solutions for big data. It defines big data as large, complex datasets that are difficult to process using traditional systems. It also describes the 3Vs of big data: volume, velocity, and variety. The document then outlines Microsoft's offerings for big data including HDInsight, .NET SDK for Hadoop, ODBC driver for Hive, and integrations with Excel, SharePoint, and SQL Server. It provides overviews of Hadoop, HDFS, MapReduce, and the Hadoop ecosystem.
Introduction to Big Data
- 1. Introduction to Big Data Joey Li joeylicc@gmail.com @joeylicc joeylicc.wordpress.com
- 2. What is Big Data? Big Data is a collection of data sets so large and complex that it becomes difficult to process using traditional database systems. Big Data Challenges (3Vs) Volume Amount of Data Velocity Speed of Data In & Out Variety Range of Data Types & Sources
- 3. Microsoft Solution to Big Data ● ● ● ● ● Microsoft HDInsight Microsoft .NET SDK for Hadoop Microsoft ODBC Driver for Hive Microsoft Excel (Power View & PowerPivot) Microsoft SharePoint (Power View)
- 4. Microsoft HDInsight ● 100% Apache Hadoop compatible Big Data implementation ● Microsoft support of HDInsight on Windows Server and Windows Azure ● Simplified deployment and ease of manageability with System Center 2012 or Windows Azure ● Elegant connectivity to Microsoft Office Excel 2013 and Business Intelligence tools
- 5. What is Hadoop? Apache Hadoop is an open-source software framework that allows for the distributed processing of large data sets across clusters of computers using simple programming model. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.
- 6. What is Hadoop? (Cont.) Hadoop includes 2 major modules 1. Hadoop Distributed File System (HDFS) A distributed file system that provides high-throughput access to application data 2. Hadoop MapReduce A programming model for parallel processing of large data sets
- 11. MapReduce
- 12. Hadoop Ecosystem
- 13. Microsoft .NET SDK for Hadoop ● ● ● ● HDInsight Cluster Management Hadoop Job Submission Customize Map/Reduce Job LINQ to Hive
- 14. Microsoft ODBC Driver for Hive ● Connect the following tools to Hadoop for data insight ○ Microsoft Excel (Power View & PowerPivot) ○ Microsoft SharePoint (Power View) ○ Microsoft SQL Server ■ Database Engine ■ Analysis Services
- 15. Learning Hadoop ● Get Started with Hadoop@Hortonworks http://hortonworks.com/get-started/ ● Big Data University http://bigdatauniversity.com/ ● Getting Started with Microsoft Big Data http://www.microsoftvirtualacademy.com/training-courses/getting-startedwith-microsoft-big-data
- 16. References ● Big Data@Wikipedia http://en.wikipedia.org/wiki/Big_data ● Big Data@Microsoft http://www.microsoft.com/en-us/sqlserver/solutions-technologies/businessintelligence/big-data.aspx ● Hortonworks Data Platform (HDP) http://hortonworks.com/
- 17. References (Cont.) ● Apache Hadoop http://hadoop.apache.org/ ● Apache Hadoop@Wikipedia http://en.wikipedia.org/wiki/Apache_Hadoop ● Microsoft .NET SDK for Hadoop http://hadoopsdk.codeplex.com/ ● Microsoft ODBC Driver for Hive http://www.microsoft.com/en-us/download/details.aspx?id=37134