Lift SSIS package to Azure Data Factory V2
- 1. Lift SSIS packages to Azure using Data Factory v2 Manjeet Singh, Consultant – ASG Group
- 3. Manjeet Singh Information & Data Management Consultant ASG Group Information and Data Management Consultant at ASG Group with more than 15 years' experience in Information & Data Management domain. Experience of working in industries such as Energy, Oil and Gas, Banking, Local Government and IT services. An aspiring Data Scientist and passionate about Artificial Intelligence (AI) and Machine Learning (ML). Place your photo here /manjeetsingh-aibipro @manjeetsinghitp
- 4. Thanks to all sponsors
- 5. Agenda
- 6. Azure Data Factory (ADF) Azure Data Factory is a cloud-based data integration service that allows you to create data-driven workflows in the cloud for orchestrating and automating data movement and data transformation.
- 7. Azure Data Factory V2.0 Everything done in Azure Data Factory v2 will use the Integration Runtime engine. The IR is the core service component for ADF v2. With the IR in ADF v2 makes it possible to lift and shift our existing on premises SSIS packages into the cloud. Or start with a blank canvas and create cloud based scale out control flow and data flow pipelines, facilitated by the new capabilities in ADFv2
- 8. Integration Runtime The Integration Runtime (IR) is the compute infrastructure used by Azure Data Factory to provide the data integration capabilities across different network environments. https://docs.microsoft.com/en-us/azure/data-factory/concepts-integration-runtime
- 9. Difference ADF v/s ADF v2 Azure Data Factory Azure Data Factory V2 Used Data Management Gateway as self hosted. Uses Integration Runtime which can be SelfHosted or Managed No SSIS execution Integration runtime can execute SSIS in addition to Moving data and executing ADF activities Monitor issues at high level only Rich logs with following metrics: • Successful pipeline runs. • Failed pipeline runs. • Successful activity runs. • Failed activity runs To make a REST call a custom activity was required and we needed C# for the interface interaction Web activity type is simply a REST API caller. Use JSON with child attributes
- 10. Deploy SSIS packages to ADF v2 Prerequisites and resources required: 1. Azure subscription 2. Azure SQL Database Server 3. Azure Data Factory v2 4. Azure-SSIS Integration Runtime
- 11. Deploy SSIS packages to ADF v2 Azure SQL Database Server should be provision in the same region as Integration Runtime. It hosts SSIS Catalog database (SSISDB)
- 12. DEMO
- 13. Create IR for ADF v2
- 14. Login to SSISDB
- 16. SSIS package deployment (continues . .)
- 17. Run deployed SSIS package • Run the deployed package using SQL Server Management Studio • Run the deployed package using a pipeline in ADF v2
- 18. Create ADF v2 Pipeline
- 19. Execute ADF v2 Pipeline
- 21. Tips/Info on using SSIS IR (Integrationruntime)
- 23. Thank you! Manjeet Singh Information and Data Management Consultant, ASG Group /manjeetsingh-aibipro @manjeetsinghitp