DatamindsConnect2019 Azure Key Vault, Azure Dev Ops and Azure Data Factory how do these Azure Services work perfectly together!
- 1. Erwin de Kreuk Microsoft Solution Architect Azure Key Vault, Azure Dev Ops and AzureDataFactory how do these Azure Services work perfectly together! Date: October 8th Room: Alcazar Time: 16:45
- 4. WHAT IS AZURE DATA FACTORY? Hybrid data integration service With visual tools, you can build, debug, deploy, operationalize and monitor your (big) data pipelines Provides a way to transform data at scale without any coding required. ELT Platform
- 5. “DevOps is the union of people, process, and products to enable continuous delivery of value to your end users” Donovan Brown
- 6. WHAT IS AZURE DEVOPS
- 8. WHAT IS AZURE KEYVAULT? The safehouse to safeguard the cryptographic keys and secrets that are used by your applications, servers and cloud applications Fully integrated with Azure Active Directory Secret management: Securely store and tightly control access to tokens, passwords, certificates, API keys, and other secrets. Key management: Create and control encryption keys that encrypt your data. Certificate management: Provision, manage, and deploy public and private Secure Sockets Layer/Transport Layer Security (SSL/TLS) certificates for use with Azure and your internal connected resources.
- 9. ROLES IN AZURE KEYVAULT? Deploys Application Tells de application the URI of the key / secret Configures the application to use key / secret (and may abuse), but never sees the keys AuditorDeveloperSecurity Operations Manages Keys Creates a Key Vault in Azure Add keys / secrets to the Key Vault Grants Permission to specific applications to perform specific operations using keys Enables usage logs Monitors access to keys Reviews usage logs to confirm proper key use and compliance with data security standards
- 10. CREATING YOUR AZURE DATA FACTORY demoedkazudvlmadfbicc01 demoedkazutestadfbicc01 Create a Data Factory for every environment Define naming conventions customername-azu-environment-azureservice-department $SubscriptionName = “XXXXXXXXXXXXXXXXX" $ResourceGroupName = "DEMOEDK-EUW-BICC-DVLM-RG-ENVIRONMENT" $DataFactoryNameName="demoedkazudvlmadfbicc01" $Location="westeurope" Connect-AzAccount Select-AzSubscription -SubscriptionName $SubscriptionName New-AzDataFactoryV2 -Name $DataFactoryNameName - ResourceGroupName $ResourceGroupName -Location $Location
- 11. SET UP CODE REPOSITORY
- 12. SET UP CODE REPOSITORY Repository Settings
- 13. AZURE DATA FACTORY Create linked Services: Azure SQL DB LS_ASQL_ENV01_WWI Azure Blob Storage LS_ABLB_ENV01 Directly saved in Azure DevOps !!
- 14. AZURE DATA FACTORY Create linked Services: Azure SQL DB LS_ASQL_ENV01_WWI Azure Blob Storage LS_ABLB_ENV01 Directly saved in Azure DevOps !! More naming conventions check: https://erwindekreuk.com/2019/04/azure-data-factory-naming-conventions/
- 15. AZURE DATA FACTORY Create Datasets: DS_ABLB_ENV01 DS_ASQL_ENV01_WWI
- 16. AZURE DATA FACTORY Create Pipeline: PL_COPY_WWI_ABLB DS_AQSL _ENV01_ WWI DS_ABLB _ENV01
- 18. CREATING YOUR AZURE KEY VAULT demoedkazudvlmadfbicc01 demoedkazutestadfbicc01
- 19. CREATING YOUR AZURE KEY VAULT Demoedkazudtapkeyvault01 $SubscriptionName = “XXXXXXXXXXXXXXXXX" $ResourceGroupName = "DEMO-EUW-DTAP-RG-ENVIRONMENT" $KeyvaultName="Demoedkazudtapkeyvault02" $Location="West Europe" Connect-AzAccount Select-AzSubscription -SubscriptionName $SubscriptionName New-AzKeyVault -Name $KeyvaultName -ResourceGroupName $ResourceGroupName -location $Location
- 20. CONNECTING KEYVAULT IN ADF Create a new Linked Service
- 21. CONNECTING KEYVAULT IN ADF Grant the Azure Data Factory access to your Azure KeyVault. Be aware that you need to publish the Azure KeyVault before you start using secrets !
- 22. CREATING SECRETS IN AZURE KEY VAULT DVLM-ABLB-ENV01 TEST-ABLB-ENV01 DVLM-ASQL-ENV01-WWI TEST-ASQL-ENV01-WWI The use of naming conventions ensures that we can configure everything more dynamically in the release process later on ! ENVIRONMENT-CONNECTION-REFERENCE Environment Development DVLM Test TEST Acceptance ACPT Production PROD Linked Service Azure Blob Storage ABLB Azure Cosmos DB SQL API ACSA Azure Cosmos DB MongDB API ACMA Azure Data Explorer ADEX Azure Data Lake Storage Gen1 ADLS Azure Data Lake Storage Gen2 ADLS Azure Database for MariaDB AMDB Azure Database for MySQL AMYS Azure Database for PostgreSQL APOS Azure File Storage AFIL Azure Search ASER Azure SQL Database ASQL Azure SQL Database Managed Instance ASQM Azure SQL Data Warehouse ASDW Azure Table Storage ATBL SQL Server MSQL Reference DatabaseConnection LogicalServer(Short)-DatabaseName Storage StorageName like Audit/SSIS/Staging/DataLake
- 24. CREATING YOUR AZURE DATA FACTORY demoedkazudvlmadfbicc01 demoedkazutestadfbicc01
- 25. CREATING LIBRARY IN AZURE DEV OPS Configure the correct values
- 26. CREATE RELEASE IN AZURE DEV OPS
- 27. CREATE RELEASE IN AZURE DEV OPS Create a Release based on an Empty Job
- 28. CREATE RELEASE IN AZURE DEV OPS Create a Release based on an Empty Job Create a Stage for each Environment Test Acceptance Production
- 29. CREATE RELEASE IN AZURE DEV OPS Create a Release based on an Empty Job Create a Stage for each Environment Add an Artifact
- 30. CREATE RELEASE IN AZURE DEV OPS Create a Release based on an Empty Job Create a Stage for each Environment Add an Artifact Enable Continuous Deployment
- 31. CREATE RELEASE IN AZURE DEV OPS Create a Release based on an Empty Job Create a Stage for each Environment Add an Artifact Enable Continuous Deployment
- 32. CREATE TASK IN RELEASE IN AZURE DEV OPS Define an Agent
- 33. CREATE TASK IN RELEASE IN AZURE DEV OPS Define an Agent Add the task
- 34. CREATE TASK IN RELEASE IN AZURE DEV OPS Define an Agent Add the task Configure the task
- 35. CREATE TASK IN RELEASE IN AZURE DEV OPS Define an Agent Add the task Configure the task Override the Template Parameters
- 36. RELEASE IN AZURE DEV OPS Connecting Library Release name format
- 42. CREATING YOUR AZURE DATA FACTORY demoedkazudvlmadfbicc01 demoedkazutestadfbicc01
- 43. CUSTOM PARAMETERS Not all the properties have parameters by default. arm-template-parameters-definition.json Adding accountkey for DataLake with secrets from Azure Key Vault “accountkey": “|", Download file from: https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#default-parameterization- template = means keep the current value as the default value for the parameter. - means don't keep the default value for the parameter. | is a special case for secrets from Azure Key Vault for connection strings or keys.
- 44. REMARKS You cannot publish while SSIS Integration Runtime is running Triggers needs to be disabled and enabled afterwards Every time you change something to your connection you need to update the Template parameter Deletes are handled manually or through Custom PS script or Marketplace extension You may run into the Azure Resource Manager template limits as your factory grows bigger. => Linked Resource Manager templates usually have a master template and a set of child templates linked to the master. Resource: https://docs.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment#linked-resource-manager- templates
- 45. HANDLING DELETES Create a post and pre deployment step in your Release
- 47. Erwin de Kreuk Microsoft Solution Architect @erwindekreuk https://erwindekreuk.com
- 48. What do you think? 10/8/2019Datamindstestpresentation 4 8 http://bit.ly/dataMindsConnectSession 1.Open the form 2.Provide constructive feedback 3.Be eligible for an amazing prize! bit.ly is CASE SENSITIVE!
- 49. Erwin de Kreuk Microsoft Solution Architect @erwindekreuk https://erwindekreuk.com