Data Factory in Microsoft Fabric (MsBIP #82)
1 like441 views
The document discusses Microsoft Fabric's data factory, emphasizing its end-to-end analytics capabilities, including no-code/low-code integration and seamless data orchestration through pipelines. It highlights features such as diverse data ingestion, transformation functionalities, and extensive connector support for various sources and destinations. Additionally, it outlines the evolution of the data factory and its integration with Power BI and Azure services.
Data Factory in Microsoft Fabric (MsBIP #82)
- 1. Data Factory in Microsoft Fabric Cathrine Wilhelmsen • MsBIP #82 • April 15th, 2024
- 4. What is MicrosoftFabric? End-to-end Analytics: • Software-as-a-Service • Lake-centric architecture • Role-tailored tools
- 5. Data Factory Data Engineering Data Warehouse Data Science Real-Time Analytics Power BI Data Activator OneLakestoring onecopy ofall yourdata inanopenformat Built-insecurity, governanceand compliance
- 6. Data Factory
- 7. The Evolution of Data Factory 2014 2017 2019 2023 2015
- 8. What is Data Factory in Fabric? No-code / low-code data integration experience with: • Scale and power of Azure Data Factory • Ease-of-useof Power Query • Intelligenceof Copilot
- 9. Ingest Data Transform Data
- 10. Orchestrate Workflows Ingest Data Transform Data
- 11. Orchestrate Workflows Ingest Data Transform Data
- 12. Orchestrate Workflows Pipelines Ingest Data Dataflows / Pipelines Transform Data Dataflows
- 13. Orchestrate Workflows Pipelines Ingest Data Dataflows / Pipelines Transform Data Dataflows
- 14. Pipelines
- 15. What are Pipelines? Pipelines are what you execute or run They define workflows: what to do in which order
- 16. What are Activities? Activities are individual steps inside pipelines They can be chained or run in parallel
- 17. Copy Data Activity Source Destination
- 18. Copy Data Activity: Complex Data Source Convert File Formats Convert Data Types Map Columns Flatten Hierarchies Split & Merge Files Compress & Decompress Destination
- 19. What are Connections? Connections define how to connect and authorize Choose between built-in connection to workspace or 36+ external sources
- 20. What are Runs? Runs are individual executions of pipelines Run on-demand or on a defined schedule
- 21. Let’s demo!
- 22. Orchestrate Workflows Pipelines Ingest Data Dataflows / Pipelines Transform Data Dataflows
- 23. Orchestrate Workflows Pipelines Ingest Data Dataflows / Pipelines Transform Data Dataflows
- 24. Dataflows
- 25. What are Dataflows (Gen2)? Power Query scaled up and out: • No-code / low-code transformations • 160+ source connectors • 300+ transformation functions • New destinations in Microsoft Fabric and Azure • New Fast Copy capability
- 26. Let’s just demo!
- 27. Decision Guide for Ingest and Transform Copy Activity Dataflow Use cases: Data Ingestion Data Conversion Data Ingestion Data Transformation Userinterface: Wizard,Canvas Power Query Sources: 30+ connectors 160+connectors Destinations: 18+connectors Workspace:Lakehouse / Warehouse External:Azure SQL Database,Azure Data Explorer,Azure SynapseAnalytics Transformations: Lightweight conversions 300+ transformationfunctions
- 28. Questions?
- 29. Thank you! hi@cathrinew.net cathrinew.net @cathrinew