Trivadis Azure Data Lake
1 like211 views
The document discusses the transformation of organizations towards cloud-first strategies, highlighting the rapid growth of data and the significant increase in AI investments. It emphasizes the importance of a modern data estate, including data lakes, to manage various data types effectively for analytics and machine learning. Additionally, it outlines Microsoft's tools and services that facilitate data migration and business intelligence solutions in this evolving landscape.
![Data lake
From Wikipedia, the free encyclopedia
A data lake is a method of storing data within a system or repository, in its natural format,[1] that
facilitates the collocation of data in various schemata and structural forms, usually object blobs or files.
The idea of data lake is to have a single store of all data in the enterprise ranging from raw data (which
implies exact copy of source system data) to transformed data which is used for various tasks including
reporting, visualization, analytics and machine learning. The data lake includes structured data from
relational databases (rows and columns), semi-structured data (CSV, logs, XML, JSON), unstructured
data (emails, documents, PDFs) and even binary data (images, audio, video) thus creating a centralized
data store accommodating all forms of data.
James Dixon / Pentaho (2010)
Data Lakes](https://image.slidesharecdn.com/trivadisdatalakeevent02032018-180305101815/85/Trivadis-Azure-Data-Lake-12-320.jpg)
Trivadis Azure Data Lake
- 1. The world is changing
- 2. Agenda - New challenges and ways - On-premises or cloud (or both?) - The union of all - Making sense of it - How to get there...
- 3. Today, 80% of organizations adopt cloud-first strategies AI investment increased by 300% in 2017 Data will grow to 44 ZB in 2020
- 4. Today, 80% of organizations adopt cloud-first strategies AI investment increased by 300% in 2017 Data will grow to 44 ZB in 2020 C LO U D A IDATA
- 5. C LO U D DATA A I Organizations that harness data, cloud, and AI outperform
- 6. Rely on a modern data estate
- 7. Patrik Borosch TSP / DP • 02.07.1971/married/Daughter(19) • Music(Squared Circle/The Midcrise Liars)/climbing/skiing/cycling/Tango • EDV-Kaufmann (german IHK 1995) = Data Processing with Cobol and Databases = the early BI guys ;) • Reporting and data processing in controlling departments • BI Consultant/Senior Consultant: ASTECH Solutions/T-Systems/Trivadis/Avanade • Discipline Manager Microsoft BI and Power Pivot Trainer @ Trivadis • Head of BI: Allianz Global Assistance • TSP DP: Microsoft • SQL Server/SSIS/SSAS/SSRS/MDM • PowerBI/PowerPivot/PowerQuery • Azure SQL DB/DW/Data Lake/Azure Stream Analytics/Data Factory/AAS • SQL/DAX/MDX/(PowerShell)/(C#) • Informatica/Microstrategy/Essbase/Enterprise Architect/UML/Perl/Unix/Linux had that... been there... • First «Big Data»-Project in 2006: Teradata/Informatica/Microstrategy, 1.4TB = eight weeks for init load = lots of fun :D
- 8. 8
- 9. 9
- 10. Dr. John Snow (1854) One of the first visual investigations of collected data helped to solve a cholera epidemic in Soho… - You need the facts - BUT you also need to make sense of it... Therefore you need to have the right tools and methods...
- 11. The many sources and rapid growth of data requires a new approach • Sentiment Analysis • Social Media / Sales Connection • Customer Segmentation
- 12. Data lake From Wikipedia, the free encyclopedia A data lake is a method of storing data within a system or repository, in its natural format,[1] that facilitates the collocation of data in various schemata and structural forms, usually object blobs or files. The idea of data lake is to have a single store of all data in the enterprise ranging from raw data (which implies exact copy of source system data) to transformed data which is used for various tasks including reporting, visualization, analytics and machine learning. The data lake includes structured data from relational databases (rows and columns), semi-structured data (CSV, logs, XML, JSON), unstructured data (emails, documents, PDFs) and even binary data (images, audio, video) thus creating a centralized data store accommodating all forms of data. James Dixon / Pentaho (2010) Data Lakes
- 13. BUSINESS APPS CUSTOM APPS ANALYTICAL DASHBOARDS AZURE SQL DATA WAREHOUSE AZURE CLI AZURE DATA FACTORY BCP COMMAND LINE UTILITY SQL SERVER INTEGRATION SERVICES AZURE ANALYSIS SERVICES
- 14. BUSINESS APPS CUSTOM APPS ANALYTICAL DASHBOARDS DATA FACTORY ANALYTICAL DASHBOARDS Polybase AZURE SQL DATA WAREHOUSE DATA FACTORY AZURE ANALYSIS SERVICES AZURE MACHINE LEARNING & MACHINE LEARNING SERVER AZURE COSMOS DB AZURE STORAGE
- 15. BUSINESS APPS CUSTOM APPS ANALYTICAL DASHBOARDS DATA FACTORY ANALYTICAL DASHBOARDS Polybase AZURE SQL DATA WAREHOUSE DATA FACTORY AZURE ANALYSIS SERVICES AZURE MACHINE LEARNING & MACHINE LEARNING SERVER AZURE COSMOS DB AZURE HDINSIGHT (Hadoop)AZURE STORAGE
- 16. BUSINESS APPS CUSTOM APPS ANALYTICAL DASHBOARDS DATA FACTORY ANALYTICAL DASHBOARDS Polybase AZURE SQL DATA WAREHOUSE DATA FACTORY AZURE ANALYSIS SERVICES AZURE MACHINE LEARNING & MACHINE LEARNING SERVER AZURE COSMOS DB AZURE STORAGE AZURE DATABRICKS (SPARK)
- 17. BUSINESS APPS CUSTOM APPS ANALYTICAL DASHBOARDS DATA FACTORY AZURE DATA LAKE STORE AZURE DATA LAKE ANALYTICS ANALYTICAL DASHBOARDS Polybase AZURE SQL DATA WAREHOUSE DATA FACTORY AZURE ANALYSIS SERVICES AZURE MACHINE LEARNING & MACHINE LEARNING SERVER AZURE COSMOS DB
- 19. CONTROL EASE OF USE Azure Data Lake Analytics Azure Data Lake Store Azure Storage Any Hadoop technology, any distribution Workload optimized, managed clusters Data Engineering in a Job-as-a-service model Azure Marketplace HDP | CDH | MapR Azure Data Lake Analytics IaaS Clusters Managed Clusters Big Data as-a-service Azure HDInsight Frictionless & Optimized Spark clusters Azure Databricks BIGDATA STORAGE BIGDATA ANALYTICS ReducedAdministration K N O W I N G T H E V A R I O U S B I G D A T A S O L U T I O N S Drag & Drop Azure ML
- 20. Big Data is driving transformative changes Cost Culture Data Characteristics Traditional Big Data Relational (with highly modeled schema) All Data (with schema agility) Expensive (storage and compute capacity) Cloud (storage and compute capacity) Rear-view reporting (using relational algebra) Intelligent action (using relational algebra AND ML, graph, streaming, image processing)
- 21. Cognitive Services • Faces, images, emotion recognition and video intelligence • Spoken language processing, speaker recognition, custom speech recognition • Natural language processing, sentiment and topics analysis, spelling errors • Complex tasks processing, knowledge exploration, intelligent recommendations • Bing engine capabilities for Web, Autosuggest, Image, Video and News Intelligence Cortana Bot Framework Cognitive Services
- 22. Microsoft BI, the agile way… Azure Analysis Services
- 24. Data Sources Ingest Prepare Analyze Publish Consume Sensors and devices Stream Analytics Diagnostic Streaming Power BI Sources - Oralce HFS - SAP BW - … Azure Data Lake Store Data Factory: Move data, orchestrate, schedule and monitor Azure Data LakeIoT Hubs Machine Learning HDInsight Data Science Workbench Stream Analytics Power BI Report Server Architecture Blueprint SSIS SQL Server 2017: Security, Performance, Polybase, ML Services, Analytics SQL Server 2017 SSAS BI Bot Apps Lab- and other Apps AzureDataPlatformSQLserver2017
- 25. AI built-in | Most secure | Lowest TCO M I C R O S O F T F O R Y O U R M O D E R N D A T A E S T A T E Data warehouses Data lakes Operational databases Data warehouses Data lakes Operational databases SQL Server Azure Data Services Industry leader 4 years in a row #1 TPC-H performance T-SQL query over any data 70% faster 2x the global reach 99.9% SLA HYBRID Easiest lift and shift with no code changes SocialLOB Graph IoTImageCRM Security and performance Flexibility of choice Reason over any data, anywhere
- 26. Tools for your migration journey SQL Server Migration Assistant (SSMA) Automates database migration to SQL Server from Microsoft Access, DB2, MySQL, Oracle, and SAP ASE. Data Migration Assistant (DMA) Enables upgrade to SQL Server and Azure SQL Database. Database Experimentation Assistant (DEA) Assists in evaluating a targeted version of SQL for a given workload. Azure Hybrid Benefit for SQL Server Maximizes current on-premises license investments to facilitate migration. Azure SQL Database Managed Instance Facilitates lift and shift migration from on-premises SQL Server to PaaS. Azure Database Migration Service (Azure DMS) SQL Server Migration Assistant (SSMA) Data Migration Assistant (DMA) Database Experimentation Assistant (DEA) SQL Database Managed Instance Azure Hybrid Benefit for SQL Server
- 27. Empower today’s innovators to unleash the power of data and reimagine possibilities that will improve our world