Microsoft Build 2018 Analytic Solutions with Azure Data Factory and Azure SQL Data Warehouse
Download as PPTX, PDF3 likes883 views
Azure Data Factory is a fully-managed cloud data integration service that offers serverless scalability, a drag-and-drop UI, and codeless data movement. It allows users to modernize their data warehouses at scale and integrates with various Azure services for data transformation and orchestration. The platform supports secure data movement and is compatible with leading BI and data integration services, enhancing performance and scalability.
Microsoft Build 2018 Analytic Solutions with Azure Data Factory and Azure SQL Data Warehouse
- 6. A fully-managed data integration service in the cloud A Z U R E D A TA F A C T O R Y H Y B R I D S C A L A B L EP R O D U C T I V E T R U S T E D Serverless scalability with no infrastructure to manage Drag & Drop UI Codeless Data Movement Orchestrate where your data lives Lift SSIS packages to Azure Certified compliant Data Movement
- 7. Modernize your enterprise data warehouse at scale A Z U R E D A T A F A C T O R Y Social LOB Graph IoT Image CRM STORE Azure Data Lake and Azure Storage MODEL & SERVE Azure SQL DW HDInsight Data Lake PREP & TRANSFORM Data Transformation Machine Learning INGEST Data orchestration, scheduling and monitoring Apps and Insights Integrate via Azure Data Factory Cloud VNet On-premise
- 8. EXTRACT TRANSFORM LOAD MODEL & SERVE Drought Weather Counties NOAA Azure Analysis Services Azure SQL Data Warehouse Data Marts for ML Azure Databricks Azure Data Factory Power BI Polybase Data Transformation Apps and Insights
- 9. Pipeline Activity Triggers Pipeline Runs Activity Runs Trigger Runs Linked Service Dataset Data Movement Data Transformation Dispatch Integration Runtime Activity Activity
- 17. Secure. Compliant. Reliable.Unlimited scale Seamlessly compatible across Microsoft and other leading BI & Data Integration ser vices The fast, flexible, and secure hub for all your data TrustedFast Fits your needs Flexible
- 18. Introducing SQL Data Warehouse Gen2 5x improvement in performance 4x concurrent queries 5x scalability Retains all elastic functionality
- 22. http://azure.com/adf http://azure.com/sqldw @AzureSqlDw #AzureDataFactory @KromerBigData ADF Github code: https://github.com/kromerm/adfbuild2018 Get Started with this ADW/ADF/Spark Template https://gallery.azure.ai/Tutorial/Data-Warehousing-and-Data-Science-with-SQL-Data-Warehouse-and-Spark-3
Editor's Notes
- #4: Start: Matt
- #6: Start: Mark
- #10: A data factory can have one or more pipelines. A pipeline is a logical grouping of activities that together perform a task. The activities in a pipeline define actions to perform on your data. For example, you might use a copy activity to copy data from an on-premises SQL Server to Azure Blob storage. Then, you might use a Hive activity that runs a Hive script on an Azure HDInsight cluster to process data from Blob storage to produce output data. Finally, you might use a second copy activity to copy the output data to Azure SQL Data Warehouse, on top of which business intelligence (BI) reporting solutions are built. A dataset is a named view of data that references the data and the structure of the data you want to use in your activities as inputs and outputs. Datasets identify data within different data stores, such as tables, files, folders, and documents. For example, an Azure Blob dataset specifies the blob container and folder in Blob storage from which the activity should read the data. Before you create a dataset, you must create a linked service to link your data store to the data factory. Linked services are much like connection strings, which define the connection information needed for Data Factory to connect to external resources. Think of it this way; the dataset represents the structure of the data within the linked data stores, and the linked service defines the connection to the data source. For example, an Azure Storage linked service links a storage account to the data factory. An Azure Blob dataset represents the blob container and the folder within that Azure storage account that contains the input blobs to be processed.