Azure Data Factory Data Wrangling with Power Query
Download as PPTX, PDF0 likes498 views
Azure Data Factory now allows users to perform data wrangling tasks through Power Query activities, translating M scripts into ADF data flow scripts executed on Apache Spark. This enables code-free data exploration, preparation, and operationalization of Power Query workflows within ADF pipelines. Examples of use cases include data engineers building ETL processes or analysts operationalizing existing queries to prepare data for modeling, with the goal of providing a data-first approach to building data flows and pipelines in ADF.
Azure Data Factory Data Wrangling with Power Query
- 1. Azure Data Factory: Data Wrangling Power Query in ADF Updated Public Preview Q1 CY21
- 2. What is Data Wrangling? Code-free data exploration and data prep Operationalize Power Query as an activity by translating M script in ADF data flow script Execute Power Query as a pipeline activity using the ADF data flow serverless, scaled-out, ADF-managed Apache Spark engine Essentially acts as a data-first entry point to building ADF data flows
- 3. ADF Data Wrangling Use Cases Data Engineer is building an ETL process in ADF uses PQ to explore data using data profiling Business Analyst is a PQ desktop user and wishes to operationalize their M query in a data pipeline that sinks data in the Lake Data Engineer needs to prep data for modeling and ETL by using a data-first approach. Creates a PQ wrangling activity and adds it to pipeline. Trimming strings Data type conversions Rename columns Remove columns Value prop “Data Wrangling in ADF”, not “Power Query lift-and-shift”
- 4. ADF Data Wrangling Roadmap – PQ Activity Continue to add more M functions to fold into Spark Add more native connectors that work in both ADF & Power Query Enable V-Net in Power Query Online data wrangling experience in ADF Launch PQ activity in Synapse Pipelines Enable interactive monitoring similar to Copy and Data Flow
- 5. Additional resources Documentation List of tutorial videos Expression language reference Performance guide ADF twitter ADF tech community blog