![What is Azure Data Factory (ADF)?
• "[Azure Data Factory] is a cloud-based data integration service that
allows you to create data-driven workflows in the cloud that
orchestrate and automate data movement and data transformation.“
• In short - it's Azure's PaaS service for time series data integration](https://image.slidesharecdn.com/introtoazuredatafactoryversion2-180221230253/85/Intro-to-Azure-Data-Factory-v1-4-320.jpg)
Intro to Azure Data Factory v1
- 1. An Introduction to Azure Data Factory v1 Everything you need to know to start developing today
- 2. Who are these people?! • Parinaz Kallick – Business Intelligence Consultant Working with BI for 10 years (Origins in databases and reporting & MSBI Stack) B.S in Computer Science MBA-IT • Eric Bragas – Business Intelligence Consultant, MCP Working with Microsoft BI for 5+ years Azure and Power BI for 3+ years California native, based in San Francisco Eastern cuisine aficionado
- 3. Agenda What is Data Factory? How does it work? Core Components How to Develop • Demo Monitoring & Management Use Cases Challenges Best Practices
- 4. What is Azure Data Factory (ADF)? • "[Azure Data Factory] is a cloud-based data integration service that allows you to create data-driven workflows in the cloud that orchestrate and automate data movement and data transformation.“ • In short - it's Azure's PaaS service for time series data integration
- 5. How Does it Work? • Leverages cloud resources to Extract, Load, and Transform your data Storage - Azure Blob Storage, HDInsight, Azure SQL DW, etc. Compute - Hive Query, Azure SQL DW, etc. • ELT over ETL • Time-series paradigm, ie. web logs, social sentiment, sensor data
- 7. Components • Pipeline - the unit of orchestration, and container for activities • Activity - a data movement or transformation component ie. Copy, HiveQuery, StoredProcedure, etc. • Linked Service - connection manager i.e. Azure Blob Storage, Azure SQL DW, etc. • Data Set - a data structure within a linked service i.e. a table or storage container, etc.
- 8. Components
- 10. Why is Data Factory Different than Other Integration Tools (*cough* *cough* SSIS) • Extract, Load, then Transform Leverage scale out compute resources to do you transforms instead of a VM running your integration service which is bound by resource limits • PaaS - pay-as-you-go Don't need a server constantly running and accruing charges • Scheduling is time-series based and implicitly defined Major paradigm shift; kind of complex initially • Built in task scheduler • Works with structured and unstructured data • Destinations are called "sinks"?
- 13. Developing Data Factories Azure Portal • Non-Microsoft clients • Exploration Visual Studio • Mature development environments • Multiple environments • Team development – easier collaboration PowerShell • Monitoring and Management • Quick setup and tear down
- 14. Demo Architecture Blog Storage - Daily Sales Files Azure SQL - Sales DataMart Data Factory Staging Table Summary Table
- 15. Demo! • Tools and extensions: Microsoft Azure Data Factory Tools for Visual Studio 2015 Cloud Explorer for Visual Studio 2015 • Spin up an Azure Data Factory Azure Storage with files and empty Azure SQL DB should be ready to go • Copy Azure Blob Storage to Azure SQL Database Use SQL write cleanup script
- 16. How do we Monitor our New Pipeline? • Azure Portal > Data Factory > Monitor & Manage • PowerShell
- 17. Use Cases • Time-series, ie. web logs, social sentiment, etc. • Hybrid integrations • Advanced Analytics workflows • Cloud migration
- 18. When ADF is NOT the Best Option • Required data sources are not supported • Loading Azure Data Warehouse Polybase is more performant • Extracting from a non-time series source • Anytime before v2 is Generally Available!
- 19. Challenges and Best Practices Challenges • The scheduling component can be very challenging to work with • The lack of expressions and variables within a control flow is a big gap Best Practices • Use consistent naming conventions • Always publish pipelines with isPaused: True • Test thoroughly before promoting to production
- 20. Azure Data Factory v2 High-level ADFv1 – is a service designed for the batch data processing of time series data ADFv2 – is a general purpose, hybrid data integration service with very flexible execution patterns New Features: • Integration Runtime (publish SSIS packages) • Branching logic (On success, On failure, On Completion, On skip) • Web Development UI • Expressions and Parameters • System Variables • Event and Scheduled Triggers • Additional Activity Types • Way more data sources! Eg. BigQuery, Dynamics 365, and way more
- 21. Fin.
Editor's Notes
- #7: All supported services in v1: https://docs.microsoft.com/en-us/azure/data-factory/v1/data-factory-create-datasets Supported Services in v2: https://docs.microsoft.com/en-us/azure/data-factory/concepts-datasets-linked-services
- #12: https://docs.microsoft.com/en-us/azure/data-factory/v1/data-factory-scheduling-and-execution
- #16: ADF Tools: https://marketplace.visualstudio.com/items?itemName=AzureDataFactory.MicrosoftAzureDataFactoryToolsforVisualStudio2015 Cloud Explorer: https://marketplace.visualstudio.com/items?itemName=MicrosoftCloudExplorer.CloudExplorerforVisualStudio2015
- #21: https://docs.microsoft.com/en-us/azure/data-factory/compare-versions https://www.purplefrogsystems.com/paul/2017/09/whats-new-in-azure-data-factory-version-2-adfv2/