ETL in the Cloud With Microsoft Azure
Download as PPTX, PDF3 likes3,087 views
The document discusses ETL (Extract, Transform, Load) processes in the context of Microsoft Azure's cloud data platform, detailing its traditional and cloud-based implementations. It describes tools like Azure Data Factory (ADF) and Azure Machine Learning (AML) for ETL and data transformation tasks, emphasizing Azure's support for various execution engines. Additionally, it highlights partner offerings in the Azure ISV marketplace, including enterprise-grade products like Informatica.
ETL in the Cloud With Microsoft Azure
- 1. Microsoft ETL in the Cloud Microsoft Azure Cloud Data Platform Mark Kromer Microsoft Azure Cloud Data Architect @kromerbigdata @mssqldude
- 2. What is ETL? • Acronym for “Extract, Transform and Load” • Classic form of data movement, aggregation, summarization, cleansing and loading a Data Warehouse • More loosely defined as data management processes that clean, move and aggregate data • Formal ETL processes are typically scheduled (i.e. hourly, nightly, monthly) • Not real-time, although micro-batch ETL systems are quite common
- 3. Classic Enterprise ETL in the Cloud with Azure Microsoft and ISV Marketplace common offerings (Examples) Spin-up SQL Server VM image from the Azure Portal to run SSIS in the cloud via Azure IaaS Informatica is an Enterprise- grade ETL product suite that offers an Azure VM available in the ISV Marketplace Microsoft partner with Azure ISV Marketplace offerings including CDC. Attunity Compose can provide additional ELT/ELT capabilities.
- 4. ELT in the Cloud with Azure Data Factory ADF provides Extract, Transform and Load in the Cloud • ADF relies on external execution engines like SQL Server, Hadoop and AzureML • Provides very easy Copy Activities to get started quickly
- 5. Azure ML as an ETL Tool Transforming Data is a common task for Data Scientists and Data Engineers • AML has a fully Cloud / Web based UI with basic SQL Transformations • AML’s core capability is training and scoring data via ML models. But you don’t need to include those advanced analytics in your “data flow”. • Schedule ETL activities via ADF Data Transformations