Building Dynamic Data Pipelines in Azure Data Factory (Microsoft Ignite 2019)
2 likes4,138 views
The document discusses the creation of dynamic solutions as an Azure Data Factory developer, focusing on the use of parameters, variables, expressions, functions, loops, and lookups. It emphasizes utilizing parameters to manage input values, executing logic in loops, and retrieving data via lookups. The content includes practical demos and examples to illustrate these concepts.
![Lookup Output: All Rows
@{activity('Lookup')
.output.value}
[
{
"Name" : "parts",
"Extension" : "csv"
},
{
"Name" : "sets",
"Extension" : "csv"
}
]](https://image.slidesharecdn.com/brk2103-cathrinewilhelmsen-azuredatafactorydynamicpipelines-191117134651/85/Building-Dynamic-Data-Pipelines-in-Azure-Data-Factory-Microsoft-Ignite-2019-49-320.jpg)
Building Dynamic Data Pipelines in Azure Data Factory (Microsoft Ignite 2019)
- 4. You are an Azure Data Factory developer
- 6. This is getting pretty tedious…
- 7. There has to be a better way!
- 9. Why would you use dynamic solutions?
- 10. How dynamic should the solution be?
- 11. How dynamic should the solution be?
- 12. What can make a solution dynamic? Parameters and Variables: Pass input values and set or update values during runtime Expressions and Functions: Modify the content of values during runtime Loops and Lookups: Control logic and executions based on external configuration values
- 15. How are Parameters passed? trigger pipeline dataset user activity linked service pipeline
- 20. How are variables controlled? Set Variable Append Variable
- 24. What are Expressions? the @ symbol "@toUpper(pipeline().parameters.FileName)" "COLORS.CSV"
- 25. What are Functions? String: Date: Collection: Logical: Conversion: Math: concat, substring, startswith, endswith … adddays, addhours, formatDateTime … co s, f s , s , g h … f, , o , qu s, ss, g … createArray, y, jso , xm , … add, sub, mul, v, m , m x, mo , …
- 27. How to combine strings? @concat() @{…}
- 28. What is String Concatenation? @concat( ' TRUNCATE TABLE dbo. ', pipeline().parameters.TableName )
- 29. What is String Interpolation? TRUNCATE TABLE dbo. @{pipeline().parameters.TableName}
- 31. Loops
- 33. Array Items
- 34. Array Items
- 36. Array Items o ch o ch o y o y o y o y o y o y @item() parts @item() colors @item() sets
- 37. Object Items
- 38. Object Items
- 39. Object Items { "Name": "colors", "Extension": "csv" } @item()
- 40. Object Items { "Name": "colors", "Extension": "csv" } @item().Name
- 41. Object Items o ch o ch o y o y o y o y o y o y @item().Name parts @item().Name colors @item().Name sets
- 42. DEMO Let's add a loop!
- 43. Lookups
- 44. Lookup
- 45. Lookup Output
- 46. Lookup Output
- 47. Lookup Output: First Row Only { "Name" : "colors", "Extension" : "csv" } @{activity('Lookup') .output.firstRow}
- 48. Lookup Output: First Row Only { "Name" : "colors", "Extension" : "csv" } @{activity('Lookup') .output.firstRow .Name}
- 49. Lookup Output: All Rows @{activity('Lookup') .output.value} [ { "Name" : "parts", "Extension" : "csv" }, { "Name" : "sets", "Extension" : "csv" } ]
- 50. Lookup Output First Row Only: Use to pass single values to another activity @{activity('Lookup').output.firstRow.ColumnName} All Rows: Use to pass entire collection to another activity @{activity('Lookup').output.value}
- 51. DEMO Let's add a lookup!
- 52. How do you make dynamic solutions? 1. 2. with parameters 3. solution in a loop 4. from configuration table
- 53. Good luck!