ADF Mapping Data Flow Private Preview Migration
0 likes673 views
The document outlines the upgrades to the GA version of Azure Data Factory (ADF) data flows, including support for managed Databricks instances and new dataset types such as CSV and Parquet. It details changes to pipeline and data flow entities, focusing on the transition from preview to GA versions and the necessary modifications for activities within pipelines. Additionally, it mentions the introduction of general compute types and core count specifications for data flows.
![Preview (Old) GA (New) Instances
{
"name":"[concat(parameters('factoryName'),
’/…')]",
"type":"Microsoft.DataFactory/factories/pip
elines",
"apiVersion":"2018-07-01-preview",
"properties":{
"activities":[
{
…
}
{
"name":"[concat(parameters('factoryName'), ’/…'
)]",
"type":"Microsoft.DataFactory/factories/pipelin
es",
"apiVersion":"2018-06-01",
"properties":{
"activities":[
{
…
}
1 change per entity
(pipeline, dataset,
dataflow)](https://image.slidesharecdn.com/dataflowga-190317221622/85/ADF-Mapping-Data-Flow-Private-Preview-Migration-5-320.jpg)
![Preview (Old) GA (New) Instances
{
"name":"[concat(parameters('factoryName'),
'/MovieDemo')]",
"type":"Microsoft.DataFactory/factories/dat
aflows",
"apiVersion":"2018-07-01-preview",
"properties":{
"sources":[
{ }
],
"sinks":[
{ }
],
"transformations":[
{ }
],
"parameters":{ }
},
"dependsOn":[ ]
}
{
"name":"[concat(parameters('factoryName'), '/Mo
vieDemo')]",
"type":"Microsoft.DataFactory/factories/dataflo
ws",
"apiVersion":"2018-06-01",
"properties":{
"type":"MappingDataFlow",
"typeProperties":{
"sources":[
{ … }
],
"sinks":[
{ … }
],
"transformations":[
{ … }
],
"parameters":{ }
}
},
"dependsOn":[ ]
}
1 change per dataflow
entity](https://image.slidesharecdn.com/dataflowga-190317221622/85/ADF-Mapping-Data-Flow-Private-Preview-Migration-6-320.jpg)
![Preview (Old) GA (New) Instances
{
"name":"[concat(parameters('factoryName'), ’/Mo
vieDemo')]",
"type":"Microsoft.DataFactory/factories/pipelin
es",
"apiVersion":"2018-07-01-preview",
"properties":{
"activities":[
{
"name":“MovieDemoDataflow",
"type":"ExecuteDataFlow",
"dependsOn":[
],
"policy":{ },
"userProperties":[ ],
"typeProperties":{
"dataflow":{ },
"linkedService":{
"referenceName":“My_Databricks",
"type":"LinkedServiceReference"
},
"staging":{ }
}
}
],
"annotations":[ ]
},
"dependsOn":[ ]
}
{
"name":"[concat(parameters('factoryName'),
’/MovieDemo')]",
"type":"Microsoft.DataFactory/factories/pi
pelines",
"apiVersion":"2018-06-01",
"properties":{
"activities":[
{
"name":“MovieDemoDataflow",
"type":"ExecuteDataFlow",
"dependsOn":[
],
"policy":{…},
"userProperties":[ ],
"typeProperties":{
"dataflow":{… },
"staging":{ },
"compute":{
"computeType": "General",
"coreCounts": 4
}
}
}
],
"annotations":[ ]
},
"dependsOn":[ ]
}
1 change per dataflow
activity in pipeline
Azure IR/Compute](https://image.slidesharecdn.com/dataflowga-190317221622/85/ADF-Mapping-Data-Flow-Private-Preview-Migration-7-320.jpg)
ADF Mapping Data Flow Private Preview Migration
- 2. • V1 • V2 – GA version • V2 w/ Data Flow (preview) – upgrading customers to the GA version
- 3. • GA version of ADF Dataflow will support • Managed Databricksinstances • New Dataflow dataset types • CSV, Parquet,DB, DW
- 5. Preview (Old) GA (New) Instances { "name":"[concat(parameters('factoryName'), ’/…')]", "type":"Microsoft.DataFactory/factories/pip elines", "apiVersion":"2018-07-01-preview", "properties":{ "activities":[ { … } { "name":"[concat(parameters('factoryName'), ’/…' )]", "type":"Microsoft.DataFactory/factories/pipelin es", "apiVersion":"2018-06-01", "properties":{ "activities":[ { … } 1 change per entity (pipeline, dataset, dataflow)
- 6. Preview (Old) GA (New) Instances { "name":"[concat(parameters('factoryName'), '/MovieDemo')]", "type":"Microsoft.DataFactory/factories/dat aflows", "apiVersion":"2018-07-01-preview", "properties":{ "sources":[ { } ], "sinks":[ { } ], "transformations":[ { } ], "parameters":{ } }, "dependsOn":[ ] } { "name":"[concat(parameters('factoryName'), '/Mo vieDemo')]", "type":"Microsoft.DataFactory/factories/dataflo ws", "apiVersion":"2018-06-01", "properties":{ "type":"MappingDataFlow", "typeProperties":{ "sources":[ { … } ], "sinks":[ { … } ], "transformations":[ { … } ], "parameters":{ } } }, "dependsOn":[ ] } 1 change per dataflow entity
- 7. Preview (Old) GA (New) Instances { "name":"[concat(parameters('factoryName'), ’/Mo vieDemo')]", "type":"Microsoft.DataFactory/factories/pipelin es", "apiVersion":"2018-07-01-preview", "properties":{ "activities":[ { "name":“MovieDemoDataflow", "type":"ExecuteDataFlow", "dependsOn":[ ], "policy":{ }, "userProperties":[ ], "typeProperties":{ "dataflow":{ }, "linkedService":{ "referenceName":“My_Databricks", "type":"LinkedServiceReference" }, "staging":{ } } } ], "annotations":[ ] }, "dependsOn":[ ] } { "name":"[concat(parameters('factoryName'), ’/MovieDemo')]", "type":"Microsoft.DataFactory/factories/pi pelines", "apiVersion":"2018-06-01", "properties":{ "activities":[ { "name":“MovieDemoDataflow", "type":"ExecuteDataFlow", "dependsOn":[ ], "policy":{…}, "userProperties":[ ], "typeProperties":{ "dataflow":{… }, "staging":{ }, "compute":{ "computeType": "General", "coreCounts": 4 } } } ], "annotations":[ ] }, "dependsOn":[ ] } 1 change per dataflow activity in pipeline Azure IR/Compute
- 8. 5) Create new Dataflow datasetsas needed