Azure Databricks (For Data Analytics).pptx
Download as PPTX, PDF1 like1,179 views
The document provides an overview of Azure Databricks, a cloud-based data and AI service by Microsoft and Databricks for analytics, machine learning, and data engineering. It outlines key features, benefits, and integration steps with Azure services, especially Azure Blob Storage, emphasizing the importance of etiquette during sessions. The document also includes practical steps for setting up and using Azure Databricks effectively.
Azure Databricks (For Data Analytics).pptx
- 2. Lack of etiquette and manners is a huge turn off. KnolX Etiquettes Punctuality Join the session 5 minutes prior to the session start time. We start on time and conclude on time! Feedback Make sure to submit a constructive feedback for all sessions as it is very helpful for the presenter. Silent Mode Keep your mobile devices in silent mode, feel free to move out of session in case you need to attend an urgent call. Avoid Disturbance Avoid unwanted chit chat during the session.
- 3. 1. What is Azure Databricks ? 2. Why we need Azure Databricks ? 3. How does Azure Databricks Works ? 4. Various Databricks. 5. Integrate Azure Databricks with Azure Blob storage.
- 5. What is Databricks Databrick was founded by original creator of Apache Spark. It was developed as a web-based platform for working with Apache Spark. It provides automated cluster management and iPython- style notebooks.
- 6. What is Databricks Azure Databricks is the jointly-developed data and AI cloud service from Microsoft and Databricks for the data analytics, data science, data engineering and machine learning.
- 7. What is Databricks Azure Databricks, architecturally, is a cloud service that lets you set up and use a cluster of Azure instances with Apache Spark installed with a Master-Worker nodal dynamic (similar to a local Hadoop/Spark cluster) Azure Cluster with Spark Remote Access
- 8. Databricks Notebooks Multi-Language Collaborative Ideal For Exploration Reproducible Get to Production Faster Enterprise Ready Adaptable
- 9. What is Databricks Since Azure Databricks is a cloud base service, it has several advantages over traditional Spark clusters. Let us look at the benefits of using Azure Databricks Optimised Spark Engine: Data Processing with Auto-scaling and Spark optimized for up to 50x performance gain. Mlfow : Track and share experiments, reproduce runs and manage models collaboratively from a central repository. Machine Learning : Pre-configured environments with frameworks such as PyTorch, TensorFlow and sci-kit learn installed.
- 10. What is Databricks Choice of language : Use your preferred language, including Python, Scala, R, Spark SQL, and .Net - whether you use serverless or provisioned computer resources.
- 11. What is Databricks Collaborative Notebooks: Quickly access and explore data and share new insights and building models collectively with the language and tools of your choice Delta Lake: Bring data reliability and scalability to your existing data lake with an open-source transactional storage layer designed for the full data cycle. Integration with Azure Services: Complete your end-to-end analytics and machine learning solution and deep integration with azure services such as Azure Data Factory, Azure Data Storage, Azure Machine learning and Power BI
- 12. What is Databricks Interactive Workspace: Easy and seamless coordination with Data Analyst Data Scientist ,Data Engineer and Business Analysist to ensure smooth collaborations. Enterprise Grade Security: The native security provided by Microsoft Azure ensure protection of data within storage services and private workspaces. Production Ready: Easily run, implement and monitor your data-oriented jobs and job-related stats.
- 13. How does Azure Databricks Works
- 14. How does Azure Databricks Works Microsoft Azure provides a very simplified and easy to use interface to implements Databricks.
- 16. Databricks Utilities Databricks utilities and DButils help us to perform a verity of powerful which include efficient object storage, chaining notebooks together and working with secrets. In Azure Databricks notebooks, the DBUtils library provides utilities for interacting with various aspects of the Databricks environment, such as file system operations, database connections, and cluster configuration. All DButils are available for notebooks of the following languages: • Python, • Scala • R Note: DBUtils are not supported outside Notebooks Overall, regardless of the notebook language you're using (Python, Scala, or R), you can leverage the capabilities provided by DBUtils to interact with various aspects of Azure Databricks.
- 17. Integrating Azure Databricks with Azure Blob Storage
- 18. Integrating Azure Databricks with Azure Blob Storage Seamless integration with various Azure services: • Azure Storage: Data storage and retrieval. • Azure SQL Data Warehouse: Data warehousing and analytics. • Azure Cosmos DB: NoSQL database for scalable applications. • Azure Data Lake Storage: Scalable data lake storage. • Azure Active Directory: Identity and access management. Microsoft azure provides a multitude of services .It often benefical to combine multiple services together to approch your use-case User Coding Notebooks Azure Databricks Azure Cluster with Spark
- 19. Hands on- integrating azure databricks with azure blob storage
- 20. Hands on- integrating azure databricks with azure blob storage Step 1: Set up Azure Databricks • Log in to the Azure portal (https://portal.azure.com). • Search for "Databricks" in the search bar. • Create a new Azure Databricks workspace by providing necessary details like subscription, resource group, workspace name, and pricing tier. • Once the workspace is provisioned, navigate to it from the Azure portal.
- 21. Hands on- integrating azure databricks with azure blob storage Step 2: Create a Cluster • Inside the Azure Databricks workspace, go to the Clusters tab. • Click on "Create Cluster" and configure the cluster settings such as cluster mode, instance type, and number of workers. • Click "Create Cluster" to provision the cluster.
- 22. Hands on- integrating azure databricks with azure blob storage Step 3: Create a Notebook • Go to the Notebooks tab in the workspace. • Click on "Create" and choose the language you want to use (Python, Scala, SQL, or R). • Name your notebook and click "Create."
- 23. Hands on- integrating azure databricks with azure blob storage Step 4: Connect to Azure Blob Storage In your notebook, use the following code to configure Azure Blob Storage credentials: pythonCopy code # Define storage account credentials storage_account_name = "your_storage_account_name" storage_account_access_key = "your_storage_account_access_key" # Configure Spark to access Azure Blob Storage spark.conf.set( "fs.azure.account.key."+storage_account_name+".blob.core.windows.net", storage_account_access_key ) Replace "your_storage_account_name" and "your_storage_account_access_key" with your actual storage account name and access key.
- 24. Hands on- integrating azure databricks with azure blob storage Step 5: Access Data in Azure Blob Storage Once connected, you can access data stored in Azure Blob Storage using Spark APIs. For example: pythonCopy code # Load data from Azure Blob Storage df = spark.read.csv("wasbs://container@storage_account_name.blob.core.windows.net/path/to/file.cs v") # Display the data display(df) Replace "container" and "path/to/file.csv" with your container name and file path.
- 25. Hands on- integrating azure databricks with azure blob storage Step 6: Perform Data Operations • You can now perform various data operations on the data loaded from Azure Blob Storage using Spark DataFrame APIs. • Analyze, transform, visualize, or model the data as needed within your notebook.
- 26. Hands on- integrating azure databricks with azure blob storage Step 7: Cleanup (Optional) • Once you're done with your analysis, you can terminate the cluster to avoid incurring unnecessary costs. • Go to the Clusters tab, select your cluster, and click "Terminate." That's it! You've successfully integrated Azure Databricks with Azure Blob Storage and performed data operations within a notebook.
- 27. Conclusion • Here we have learned about Azure Databricks • Feature of Azure Databricks • And, implementation with Blob Storage • We can explore further and leverage Azure Databricks and Azure Blob Storage for data analytics needs.