From the course: Microsoft Azure Data Engineer Associate (DP-203) Cert Prep by Microsoft Press

Unlock this course with a free trial

Join today to access over 24,700 courses taught by industry experts.

Integrate Jupyter and Python Notebooks into a data pipeline

Integrate Jupyter and Python Notebooks into a data pipeline - Azure Tutorial

From the course: Microsoft Azure Data Engineer Associate (DP-203) Cert Prep by Microsoft Press

Start my 1-month free trial Buy for my team

Integrate Jupyter and Python Notebooks into a data pipeline

“

- [Instructor] I actually mentioned this in the previous lesson. There is in Azure Data Factory a couple activities that relate to these big data products. We can bring in synapse notebooks. We can also bring in Databricks notebooks that may encapsulate an entire process themselves. For example, we might have maintained, not one, not two, but maybe multiple machine learning models. And we've encapsulated the training not in Azure Machine Learning Service, but maybe we're using Azure Databricks, Azure Synapse Spark pools. Well, we can encapsulate, as I said, all of that training logic in a notebook and then manifest those notebooks in a pipeline. So again, we keep recursively coming back and back and back to Azure Data Factory. It really is the shining star of this entire course. And then we have make sure to create linked services to your data source. Hopefully it makes sense that if you include an Azure Data Factory…

Contents