NDC Sydney - Analyzing StackExchange with Azure Data Lake
Download as PPTX, PDF0 likes186 views
The document discusses Azure Data Lake, its functionalities, and its application in analyzing data from StackExchange. It covers key features such as security, pricing, and analytics capabilities, specifically highlighting the benefits of using Azure Data Lake for big data operations. Additionally, it emphasizes the differences between data lakes and traditional data warehousing, advocating for the use of Azure Data Lake for efficient data management.
NDC Sydney - Analyzing StackExchange with Azure Data Lake
- 1. Analyzing StackExchange with Azure Data Lake Tom Kerkhove NDC Sydney 2017
- 2. Tom Kerkhove Azure Consultant @ Codit Microsoft Azure MVP & Advisor Member of Azug.be “Integration of Things” whitepaper (https://bit.ly/azure-iot) 2 Nice to meet you blog.tomkerkhove.be @TomKerkhove tomkerkhove
- 3. Agenda • Introduction to Azure Data Lake • What is Azure Data Lake Store? • What is Azure Data Lake Analytics? • Analyzing StackExchange with Azure Data Lake 3
- 4. 4
- 5. 5
- 6. Let’s go open-source, right?! ➔ Comes with a few challenges for C#/SQL professional ➔ New languages to learn & maintain ➔ Rapidly evolving ecosystem ➔ Cluster management ➔ Typically linux machines 6
- 7. 7 Analyzing Big Data in Azure
- 8. ➔ WebHDFS compatible ➔ Any size ➔ Any format as-is ➔ Write-once-read-many ➔ Enterprise-grade security ➔ Thé big data store in Azure 8 Azure Data Lake Store
- 9. 9
- 10. Characteristics ➔ Data Warehousing ➔ Structured data ➔ Defined set of schemas ➔ Requires Extract-Transform- Load (ETL) before storing ➔ Known for some of us ➔ Exploratory analysis is hard because of transforming the data 10 Data Warehousing vs Data Lakes ➔ Data Lakes ➔ Raw data (unstructured/semi-structured/structured) ➔ “Dump” all your data in the lake ➔ Data scientists will interpret data from the lake ➔ Without metadata, turns in a data swamp pretty fast
- 11. 11Martin Fowler on Data Lake & Data Warehouses: https://bit.ly/martin-fowler-data-lake
- 12. Security ➔ Roled-based Access Control (RBAC) ➔ Grant user/groups access to folder/file (https://bit.ly/data-lake-store-acls) ➔ Firewall (off by default) ➔ Encryption at rest ➔ Keys managed by Microsoft ➔ Bring-your-own-key with Azure Key Vault 12
- 13. ➔ ~$0,032/GB stored per month ➔ Transaction costs ➔ ~$0,043 per 1M write transactions ➔ ~$0,0034 per 1M read transactions ➔ 1 transaction is block of up to 128 kB ➔ Regular Egress fees ➔ Monthly commitment packages ➔ Save up to 33% 13 Pricing
- 14. Azure Data Lake Store vs Blob Storage 14 No Limitations Store whatever you want in any format Security Built-in Azure Active Directory support Pricing More expensive than Storage GRS Redundancy It’s there but no control over it Built for Scale Optimized for high- scale reads Integration With Data Factory, Data Catalog & HDInsight Full comparison on https://bit.ly/adls-vs-storage
- 15. Azure Data Lake Analytics ➔ Run analytics jobs on managed clusters ➔ No maintenance ~ Serverless ➔ Written in U-SQL ➔ SQL Syntax ➔ Extensibility in C# ➔ Easily scaled with Analytics Units ➔ Pay for processing time only 15
- 16. ➔ Built-in partitioned tables ➔ Query data where it lives ➔ No need to prepare data ➔ One query that runs on multiple data stores ➔ Use the correct data store for the job 16 Data Sources
- 17. Writing U-SQL scripts 17 Extract from data source by using built-in or custom extractors. Transform / Analyse the data using SQL-syntax, in-line C# or C# method calls Output the result to a data source by using built-in or custom extractors
- 18. ➔ C# Expressions ➔ User-Defined Functions (UDF) ➔ User-Defined Operations (UDO) ➔ User-Defined Aggregators (UDAGG) 18 Extensibility
- 19. ➔ User-Defined Extractors ➔ User-Defined Processors ➔ Take one row and produce one row ➔ Pass-through versus transforming ➔ User-Defined Reducers ➔ Take n rows and produce 1 row 19 ➔ User-Defined Outputters ➔ User-Defined Appliers ➔ Take one row and produce 0 to n rows ➔ Used with OUTER/CROSS APPLY ➔ User-Defined Combiners ➔ Combines rowsets (like a user- defined join) User-Defined Operations (UDO)
- 21. 21 U-SQL Batch Job Execution Lifetime Michael Rys on “Tuning & Optimizing U-SQL” https://bit.ly/tuning-optimizing-u-sql
- 22. 22
- 23. 23 Job States
- 24. ➔ Roled-based Access Control (RBAC) ➔ Firewall (Off by default) ➔ Access control on service catalog ➔ Access control on a per-database level 24 Security
- 25. ➔ Account-level limitations ➔ Maximum of AUs ➔ Maximum of concurrent job ➔ Days to retain queries ➔ Job-level limitations ➔ Maximum of AUs ➔ Maximum priority ➔ Granted per user and/or group 25 Resource Management
- 26. ➔ Billed for processing time, not per job ➔ Billed per second ➔ $1,687 per hour per Analytics Unit ➔ ~ $0,028 per minute ➔ Monthly commitment packages ➔ Save up to 74% 26 Pricing
- 27. Demo
- 28. Meet StackExchange ➔ Over 280 websites ➔ 150+ GB of open-source data ➔ Different kinds of data ➔ Posts ➔ Users ➔ Votes ➔ ... ➔ A big data sample data set 28
- 29. What Are We Going To Do? • Download the original data set Acquiring The Data • Upload data set to Azure • Determine what service to use Moving The Data • Merging data from each site into one file • Conversion from XML to CSV Aggregating The Data • Run business logic on it • Attempt to gain knowledge from it Analyzing The Data • Visualize what we’ve learned Visualizing The Data 29
- 30. 30 How is it setup?
- 31. ➔ Azure Data Lake Store ➔ Graphs • Storage Utilization • Read/Write • Ingress/Egress ➔ Audit & Request logs ➔ No Metrics ➔ No Alerts 31 ➔ Azure Data Lake Analytics ➔ Graphs • Job status • Used # of AU time ➔ Metrics • Job status • Used # of AU time ➔ Audit & Request logs ➔ No alerts Operations
- 32. ➔ Store Explorer ➔ Browse store ➔ Download complete / subset of file ➔ Preview ➔ Only in Visual Studio ➔ Job Visualizer ➔ Determine bottlenecks by using heatmaps ➔ Playback jobs based on telemetry ➔ Query optimization ➔ Job Profiler 32 Azure Data Lake tools for Visual Studio
- 33. ➔ Integration with Source control ➔ Unit Testing extensibility ➔ Local execution ➔ Simulate Data Lake Store ➔ Run & debug jobs 33 Azure Data Lake tools for Visual Studio (Code)
- 34. ➔ Integrate with your data pipelines in Azure Data Factory ➔ Move data from Azure Data Lake Store to other store ➔ Move data to Azure Data Lake Store ➔ Run U-SQL jobs within pipeline ➔ Integration with Azure Data Catalog ➔ Register your Azure Data Lake Store assets 34 Integration with Azure Services
- 35. ➔ Azure Data Lake Best Practices by Microsoft (Contact me) ➔ “Mastering Azure Analytics” by Zoiner Tejada (https://bit.ly/mastering-azure-analytics) ➔ MVA “Introducing Azure Data Lake” (https://bit.ly/intro-to-azure-data-lake) ➔ Azure Data Lake GitHub Repo (https://azure.github.io/AzureDataLake/) ➔ U-SQL Documentation (https://usql.io) 35 Learn more!
- 36. ➔ Big Data is not just a hype so get ready ➔ Azure Data Lake Store ➔ Analyse today & explore tomorrow ➔ Data Swamps ➔ Data Lake Analytics ➔ No cluster management ~ “Serverless” ➔ Re-use existing skills ➔ Pay for what we use ➔ Big Data in Azure? Use Azure Data Lake! 36 Summary
- 37. 37