Embrace and Extend - First-Class Activity and 3rd Party Ecosystem for SSIS in ADF

On-premises data sources SQL Server
 OS: Windows/Linux
 SCALABILITY: Scale-Out feature
 EDITION: Standard/Enterprise
 TOOLS: SSDT/SSMS to design/deploy/
manage/execute/monitor packages
 EXTENSIBILITY: ISVs can build
components/extensions on SSIS
 PRICING: Bundled w/ on-prem SQL Server

On-premises data sources
Azure SQL DB/Managed InstanceAzure Data FactoryCloud data sources
Cloud
On-premises
SQL Server
 LIFT & SHIFT: Use Azure SQL DB/Managed
Instance (MI) to host SSISDB
 SCALABILITY: Use ADF to provision a
managed cluster of Azure VMs dedicated
to run your packages – Azure-SSIS
Integration Runtime (IR)
 EDITION: Standard/Enterprise
 TOOLS: SSDT/SSMS + ADF app to
design/deploy/manage/execute/monitor
packages (activities)
 EXTENSIBILITY: ISVs can build
components/extensions + SaaS on SSIS in
ADF via custom setup + 3rd party licensing
 PRICING: Pay per hour + Azure Hybrid
Benefit (AHB) to Bring Your Own License
(BYOL)

On-premises data sources
Azure SQL DB/Managed Instance
VNet
Azure Data FactoryCloud data sources
Cloud
On-premises
SQL Server
 HYBRID: Join Azure-SSIS IR to a VNet that is
connected to your on-prem network to
enable on-prem data access
 MODERNIZATION: Schedule first-class SSIS
activities in ADF pipelines via SSMS and
chain/group them w/ other activities via
ADF app
 COMPLEMENTARY: Splice/inject built-
in/custom/Open Source/3rd party SSIS tasks
and transformations in ADF pipelines
 READINESS: General Availability (GA) w/
24/7 live-site support

Azure-SSIS IR node
Container
ISV Setup1. Specify Product Key in setup script ISV Activation Server
2. Get Activation Key by submitting Cluster ID + Product Key
Local Store
(e.g. Registry)
3. Write Activation Key
SSIS Executor
ISV Extension
4. Read Activation Key and
validate it with Cluster ID
Setup
Runtime
4. Get Cluster ID
4. Report on Node Count (Optional)
SSIS Runtime
2. Activation Key

Create ADF if you have not done
so already

Click on “Configure SSIS
Integration Runtime”

Name and describe
your Azure-SSIS IR

Select its location
 Select the right location for your Azure-SSIS IR to achieve high
performance in ETL workflows
 It needs not be the same as the location of your ADF, but it should be
the same as the location of your own Azure SQL DB/MI server where
SSISDB will be hosted and or the location of VNet connected to your
on-prem network
 Avoid Azure-SSIS IR accessing SSISDB/data across different locations

Select its node size
 A selection of node sizes specifying the number of cores (CPU) and
size of memory (RAM) per node is provided
 Select a large node size (scale up), if you want to run many compute/
memory –intensive packages

Select its cluster size
 A range of 1 to 10 for the number of nodes per cluster is provided
 Select a large cluster with many nodes (scale out), if you want to run
many packages in parallel
 Like Azure VMs, you can manage the cost of running Azure-SSIS IR
by stopping/starting it, effectively releasing/reallocating its nodes, as
you see fit

Select its edition/license
 Standard and Enterprise editions/licenses are available
 Enterprise Edition lets you use advanced/premium features on your
Azure-SSIS IR

OPTIONAL: Select to use Azure
Hybrid Benefit (AHB)
 AHB lets you Bring Your Own License (BYOL) to benefit from cost
savings with hybrid use – Essentially paying only for the infrastructure
and service management fees of your Azure-SSIS IR

Bring your own Azure SQL DB/MI
server to host SSISDB
 A selection of your existing Azure SQL DB/MI servers in the same
location as your Azure-SSIS IR is provided – If there is none, create a
new one to ensure that your Azure-SSIS IR can easily access SSISDB
without incurring excessive traffics between different locations
 Based on the selected database server, SSISDB will be created on
your behalf as a single database, part of an Elastic Pool, or in Azure
SQL MI and accessible in public network or by joining a VNet.
 The endpoint of your selected Azure SQL DB/MI server becomes the
connection info to enter on SSDT/SSMS

 If you select your existing Azure SQL MI that was provisioned inside a
VNet, you must also join your Azure-SSIS IR to the same VNet, but a
different subnet in the next provisioning blade/page, so we can
prepare and manage SSISDB on your behalf
 No need to provide any service tier for Azure SQL MI nor consent for
Azure services to access it

 If you select your existing Azure SQL DB w/ VNet service endpoints,
you must also join your Azure-SSIS IR to the same VNet and subnet
in the next provisioning blade/page, so we can prepare and manage
SSISDB on your behalf
 No need to provide consent for Azure services to access your Azure
SQL DB

 If you select your existing Azure SQL DB as Elastic Pool, you can
select your Elastic Pool name under Catalog Database Service Tier

OPTIONAL: Select to use Azure
Active Directory (AAD)
authentication
 AAD authentication w/ ADF Managed Service Identity (MSI) lets you
provision Azure-SSIS IR w/o Azure SQL DB/MI server admin
credentials
 Need to add your ADF MSI into an AAD group with access
permissions to your Azure SQL DB/MI server

Enter your Azure SQL DB/MI server
admin credentials for SQL
authentication
 Your Azure SQL DB/MI server admin username and password are
required for us to prepare and manage SSISDB on your behalf

Select the service tier and provide
access consent for your Azure SQL
DB server
 A selection of service tiers (e.g. Basic/S0/S1/etc.) for your Azure SQL
DB server is provided – Not applicable to Azure SQL MI
 Select a large service tier, if you want to achieve high database performance
and store many SSIS projects/packages/execution logs
 By default, you must allow Azure services to access your Azure SQL
DB server – Not applicable to Azure SQL MI/Azure SQL DB w/ VNet
service endpoints
 This is required for us to prepare and manage SSISDB on your behalf

Test connection to your Azure SQL
DB server
 Ensure that your Azure SQL DB server is accessible and has no
existing SSISDB, so we can prepare a new one and manage it on your
behalf

Select the concurrency level of your
Azure-SSIS IR nodes
 A range of 1 to 2 * the number of cores for the number of parallel
executions per node is provided
 Select a low concurrency level, if you want to use more than one
cores/lots of memory to run a single large (compute/memory -
intensive) package
 Select a high concurrency level, if you want to run more than one
small (light-weight) packages in a single core

OPTIONAL: Add custom setup for
your Azure-SSIS IR
 Provide a Shared Access Signature (SAS) Uniform Resource Identifier
(URI) of your Azure Storage Blob container where your setup script +
its associated files are stored

OPTIONAL: Join your Azure-SSIS IR
to a VNet and provide consent for
VNet configuration
 If you use Azure SQL MI that was provisioned inside a VNet/Azure
SQL DB w/ VNet service endpoints, you must also join your Azure-
SSIS IR to the same VNet, so we can prepare and manage SSISDB on
your behalf
 If you have on-prem data sources/destinations in your SSIS
packages, you must join your Azure-SSIS IR to a VNet that is
connected to your on-prem network to enable on-prem data access
 By default, you must allow Azure services to configure the
permissions/settings of selected VNet for your Azure-SSIS IR to join

Select VNet and subnet for your
Azure-SSIS IR to join
 A selection of existing VNets and subnets in the location of your
Azure-SSIS IR is provided – ARM VNet is recommended, since Classic
VNet will be deprecated soon
 Select the same VNet used for Azure SQL MI/Azure SQL DB w/ VNet
service endpoints
 Select a different subnet than the one used for Azure SQL MI
 Select the same subnet as the one used for Azure SQL DB w/ VNet service
endpoints
 Ideally, your Azure-SSIS IR, Azure SQL DB/MI server, and VNet
connected to your on-prem network should all be in the same
location
 If that is not the case, you can first create your Azure-SSIS IR in the location
of Azure SQL DB/MI server and join it to a VNet in the same location, then
configure a VNet-to-VNet connection with the VNet connected to your on-
prem network

When you submit a
provisioning request, it
takes 20 – 30 minutes to
allocate/prepare the
nodes of your Azure-SSIS
IR and start billing

Once your Azure-SSIS IR
is started, you can deploy
SSIS packages to execute
on it and you can stop it
as you see fit

Before stopping your
Azure-SSIS IR, you need to
consider all of its
implications

When you stop your
Azure-SSIS IR, it takes 1 – 2
minutes to release the
allocated nodes and stop
billing

Once your Azure-SSIS IR is
stopped, you can restart,
edit, or delete it

When you edit your
Azure-SSIS IR, only certain
properties are editable,
and when you finish, you
can restart it

System.Management.Automation.PSCredential($SSISDBServerAdminUserName,
$secpasswd)
}

Once your Azure-SSIS IR is
provisioned, you can
monitor it

You can click on the status
of your Azure-SSIS IR

You can start/stop your
Azure-SSIS IR and copy its
resource ID

You can click on your
Azure SQL DB/MI server
endpoint

You can copy your Azure
SQL DB/MI server
endpoint and monitor
your SSISDB

You can monitor your
pipeline runs

activity runs and rerun
your pipeline

You can obtain SSISDB
operation/execution ID
from the output of your
activity run to check on
SSMS

trigger runs

on premises
On SSMS, you can connect to SSIS in ADF
using its connection info and SQL/AAD
authentication credentials

on premises
On “Connection Properties” tab, enter
“SSISDB” as the target database to connect

on premises
in Azure
Once connected, you can deploy
projects/packages to SSIS in ADF from your
local file system/SSIS on premises

On Integration Services Deployment Wizard,
enter SSIS in ADF connection info and SQL
authentication credentials

Click on “Connections”
and ”Linked Services”

Click on “+ New” and
”Azure SQL Database”

Name and describe your
linked service

Select ADF Default Azure
IR to connect to your
Azure SQL DB that is
attached to Azure-SSIS IR

Select your Azure SQL DB
server and SSISDB to
connect

Enter your Azure SQL DB
server admin credentials

Test connection and save
your linked service

Click on “+” and ”Pipeline”

pipeline

Drag and drop “Stored
Procedure” activity into
your pipeline

sproc activity

Select your linked service
and test connection

Use system sproc
“sp_executesql” to execute
T-SQL script

Add a sproc parameter
“stmt” of type “string” with
your T-SQL script to
create/start SSIS package
execution using SSISDB
sprocs as its value

Click on “Test
Run”/“Trigger Now”

Drag and drop “Execute
SSIS Package” activity into
your pipeline

SSIS activity

Define the timeout/retry
counts/retry interval for
your SSIS activity

Select Azure-SSIS IR that is
attached to SSISDB where
your package is deployed

Check if your package
requires 32-bit runtime to
execute

Select the
predefined/customized
scope of logging for your
package execution

Provide the path to
identify your deployed
package in SSISDB, e.g.
FolderName/ProjectName
/PackageName.dtsx

Provide the path to
identify your package
execution environment in
SSISDB, e.g.
FolderName/Environment
Name, assigning values to
SSIS project/package
parameters

Click on “Validate”/“Trigger
Now”

Click on “Trigger” and
”New/Edit”

Click on “Choose trigger…”
and ”+ New”

trigger

Select a trigger of type
“Schedule”

Select the start date,
recurrence, and end date
of your trigger

Finish and publish your
trigger

Click on “Triggers” to see
your published triggers

Right-click on your
package and select
“Schedule”

ADF
pipeline/activity/trigger to
schedule your package
executions will be
automatically generated

Assign your package
execution properties and
environment

Create your package
execution schedule just
like using SQL Server
Agent on premises

You can monitor/edit your
auto-generated ADF
pipeline/activity/trigger on
ADF app

Edit your auto-generated
ADF pipeline/activity

Edit your auto-generated
ADF trigger

Create Automation account if
you have not done so already

If the latest AzureRM.Profile and
AzureRM.DataFactoryV2
modules are not listed, click on
“Browse gallery”

Search for the required modules
and import them

Click on “+ Add a runbook”

Create PSH runbook to
start/stop your Azure-SSIS IR

Enter PSH script, then save and
publish your runbook

Enter runbook parameters to
start/stop your Azure-SSIS IR

Check “Output”/”All Logs”

Click on “Schedules” and ”+ Add
a schedule”

Create a schedule for your
runbook

Click on “Webhooks” and ”+ Add
Webhook”

Create webhooks for your
runbook and save your URLs

Create ADF pipeline with a web
activity

Enter the webhook URL to start
your Azure-SSIS IR and select
POST method

Add a sproc activity to execute
your package(s)

Add a web activity to stop your
Azure-SSIS IR on completion of
the sproc activity

Trigger/schedule your pipeline

Alternatively, if you use SSIS
activity for this, specify a
sufficient number of retry
attempts that are frequent
enough to wait until your Azure-
SSIS IR is started

Enterprise Features Descriptions
CDC components • CDC Source/Splitter/Control Task are preinstalled on your Azure-SSIS IR Enterprise
Edition
• To connect to Oracle, you will also need to install CDC Designer/Service on another
machine
Oracle connectors • Oracle Connection Manager/Source/Destination are preinstalled on your Azure-SSIS
IR Enterprise Edition
• You will also need to install Oracle Call Interface (OCI) driver, and if necessary
configure Oracle Transport Network Substrate (TNS), on your Azure-SSIS IR (via
Custom Setup Interface)
Teradata connectors • You will need to install Teradata Connection Manager/Source/Destination and
Teradata Parallel Transporter (TPT) API + Teradata ODBC driver on your Azure-SSIS IR
Enterprise Edition (via Custom Setup Interface)

SAP BW connectors • SAP BW Connection Manager/Source/Destination are preinstalled on your Azure-SSIS
IR Enterprise Edition
• You will also need to install SAP BW driver on your Azure-SSIS IR (via Custom Setup
Interface)
• These connectors support SAP BW 7.0 or earlier versions
• To connect to later versions or other SAP products, you can install SAP connectors
from 3rd party ISVs on your Azure-SSIS IR (via Custom Setup Interface)

SSAS/AAS components • Data Mining Model Training/Dimension Processing/Partition Processing Destinations
and Data Mining Query Transform are preinstalled on your Azure-SSIS IR Enterprise
Edition
• All support SSAS, but only Partition Processing Destination supports AAS
• To connect to SSAS, you will also need to configure Windows Authentication
credentials in your SSISDB
• On top of these components, Analysis Services Execute DDL/Processing and Data
Mining Query Tasks are also preinstalled on your Azure-SSIS IR Standard/Enterprise
Edition
Fuzzy Grouping/Lookup
Transforms
• They are preinstalled on your Azure-SSIS IR Enterprise Edition and support SQL
Server/Azure SQL DB for storing reference data
Term Extraction/Lookup
Transforms
• They are preinstalled on your Azure-SSIS IR Enterprise Edition and support SQL
Server/Azure SQL DB for storing reference data

 If you want to use gacutil.exe to install assemblies in the Global Assembly
Cache (GAC), you need to provide it as part of your custom setup, or use
the copy provided in the Public Preview container
 If you need to join your Azure-SSIS IR w/ custom setup to a VNet, only
ARM VNet is supported

version 5.4 or later
Azure Storage Explorer app

 “Sample” folder that
contains a custom setup to
install a simple task that just
sleeps for a few seconds on
each node of your Azure-
SSIS IR – It also contains
a ”gacutil” folder, which
contains gacutil.exe, while
main.cmd contains
comments to persist access
credentials for file shares

 Double click
“UserScenarios” and you
can find
 “.NET FRAMEWORK
3.5” folder, that contains a
custom setup to install an
earlier version of the .NET
Framework on each node of
your Azure-SSIS IR
 “BCP” folder that contains a
custom setup to install SQL
Server command line utilities
(MsSqlCmdLnUtils.msi),
including bulk copy program
(bcp), on each node of your
Azure-SSIS IR
 “EXCEL” folder that contains
a custom setup to install
Open Source Excel
assemblies on each node of
your Azure-SSIS IR

To ensure that MSDTC is started, please add Execute Process Task at the beginning of control flow in
your packages to execute the following command: %SystemRoot%system32cmd.exe /c powershell -
Command "Start-Service MSDTC"
the latest Oracle
client (e.g. Oracle’s site
Oracle’s site

PSH scripts to manipulate your Azure Storage
account
Azure Blob
Download Task Execute Process Task that
executes it
Teradata’s site

sawinark@microsoft.com
https://www.yammer.com/azureadvisors/#/threads/inGroup?type=in_group
&feedId=12090139&view=all
https://blogs.msdn.microsoft.com/ssis/

https://docs.microsoft.com/en-us/azure/data-factory/tutorial-create-azure-ssis-runtime-portal
https://docs.microsoft.com/en-us/azure/data-factory/join-azure-ssis-integration-runtime-virtual-network
https://docs.microsoft.com/en-us/sql/integration-services/lift-shift/ssis-azure-connect-with-windows-auth
https://docs.microsoft.com/en-us/sql/integration-services/lift-shift/ssis-azure-connect-to-catalog-database
https://docs.microsoft.com/en-us/sql/integration-services/ssis-quickstart-deploy-ssms
https://docs.microsoft.com/en-us/sql/integration-services/ssis-quickstart-run-ssms
https://docs.microsoft.com/en-us/sql/integration-services/lift-shift/ssis-azure-schedule-packages
https://docs.microsoft.com/en-us/azure/data-factory/how-to-invoke-ssis-package-stored-procedure-activity
https://docs.microsoft.com/en-us/azure/data-factory/how-to-invoke-ssis-package-ssis-activity
https://docs.microsoft.com/en-us/azure/data-factory/monitor-integration-runtime#azure-ssis-integration-runtime
https://docs.microsoft.com/en-us/azure/data-factory/manage-azure-ssis-integration-runtime
https://docs.microsoft.com/en-us/azure/data-factory/how-to-schedule-azure-ssis-integration-runtime
https://docs.microsoft.com/en-us/azure/data-factory/how-to-configure-azure-ssis-ir-enterprise-edition
https://docs.microsoft.com/en-us/azure/data-factory/how-to-configure-azure-ssis-ir-custom-setup
https://docs.microsoft.com/en-us/azure/data-factory/how-to-develop-azure-ssis-ir-licensed-components

Embrace and Extend - First-Class Activity and 3rd Party Ecosystem for SSIS in ADF

More Related Content

What's hot (9)

Similar to Embrace and Extend - First-Class Activity and 3rd Party Ecosystem for SSIS in ADF (20)

Recently uploaded (20)

Embrace and Extend - First-Class Activity and 3rd Party Ecosystem for SSIS in ADF

Editor's Notes