Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)
0 likes116 views
Cathrine Wilhelmsen gave a presentation on using Microsoft Data Factory for data integration within Microsoft Fabric. Data Factory allows users to define data pipelines to ingest, transform and orchestrate data workflows. Pipelines contain activities that can copy or move data between different data stores. Connections specify how to connect to these data stores. Dataflows Gen2 provide enhanced orchestration capabilities, including defining activity dependencies and schedules. The presentation demonstrated how to use these capabilities in Data Factory for complex data integration scenarios.
Data Integration with Data Factory (Microsoft Fabric Day Oslo 2023)
- 1. Data Integration with Data Factory Cathrine Wilhelmsen • Microsoft Fabric Day • December 6th, 2023
- 2. Cathrine Wilhelmsen She / Her Solutions Architect, Evidi hi@cathrinew.net cathrinew.net @cathrinew I love data and coding, as well as teaching and sharing knowledge Microsoft Data Platform MVP Organizing Fabric February Renovating a house
- 3. I want to be the Chief Integration Officer
- 4. What is Microsoft Fabric? All-in-one service: • Everything out of the box • Software-as-a-Service built from the ground up • From Power BI’s user perspective
- 5. What is Data Factory? Cloud-scale data movement and orchestration service
- 6. The evolution of Data Factory 2014 2017 2019 2023 2015
- 7. Data Factory in Microsoft Fabric Data Pipelines Ingest Data Orchestrate Workflows Dataflows Gen2 Ingest Data Transform Data
- 9. What are Pipelines? Pipelines are what you execute or run Define workflows: what to do in which order
- 10. What are Activities? Activities are individual steps inside pipelines Can be chained or run in parallel
- 11. What are Connections? Connections specify how to connect and authorize to data stores and services Over 100+ connections out of the box
- 12. Can’t we just … all the things? Upload all the things? Shortcut all the things? Mirror all the things? Not always!
- 13. Copy Data Activity: Binary Files Source Sink
- 14. Copy Data Activity: Complex Data Source Sink Serialization Deserialization Compression Decompression Column Mapping
- 15. Copy Data Activity: Complex Data Source Sink Serialization Deserialization Compression Decompression Column Mapping Convert file formats
- 16. Copy Data Activity: Complex Data Source Sink Serialization Deserialization Compression Decompression Column Mapping Zip or unzip files
- 17. Copy Data Activity: Complex Data Source Sink Serialization Deserialization Compression Decompression Column Mapping Map columns implicitly or explicitly
- 18. Dataflows Gen2
- 19. What are Dataflows Gen2?
- 21. What does orchestration mean? Defining Workflows Which activities to run in which order? Configuring Alerts and Error Handling How to handle unexpected results and failures? Adding Schedules When to execute pipelines?
- 25. What are considered errors? Some events are clearly errors: Activities failing because of connectivity issues, timeouts, files not found etc. Other things might be considered errors without failing: Lookup returns unexpected data, wildcard paths only finds a subset of data, copy data activity skips many rows etc.
- 26. Triggers: Schedule Execute one or more pipelines on a set schedule: • Every Wednesday at 06:00 • Last day of the month at 18:00 • Every Monday at 04:00 and Friday at 20:00
- 27. Let’s demo!
- 28. Data Factory in Microsoft Fabric
- 29. Want to learn more about Fabric? www.fabricfebruary.com Join us February 8th, 2024 • Oslo, Norway
- 30. Thank you! hi@cathrinew.net cathrinew.net @cathrinew