Azure data bricks by Eugene Polonichko
Download as PPTX, PDF7 likes497 views
This document provides an overview of Azure Databricks, including: - Azure Databricks is an Apache Spark-based analytics platform optimized for Microsoft Azure cloud services. It includes Spark SQL, streaming, machine learning libraries, and integrates fully with Azure services. - Clusters in Azure Databricks provide a unified platform for various analytics use cases. The workspace stores notebooks, libraries, dashboards, and folders. Notebooks provide a code environment with visualizations. Jobs and alerts can run and notify on notebooks. - The Databricks File System (DBFS) stores files in Azure Blob storage in a distributed file system accessible from notebooks. Business intelligence tools can connect to Databricks clusters via JDBC
Azure data bricks by Eugene Polonichko
- 1. Azure DataBricks for Data Engineering Eugene Polonichko Senior Software Developer at Eleks, Data Platform MVP 2 0 1 8 U k r a i n e https://www.linkedin.com/in/eugenepolonichko /
- 2. About me Eugene Polonichko has over 7 years of experience with SQL Server. He mainly focused on BI projects (SSAS, SSIS, PowerBI, Cognos, Informatica PowerCenter, Pentaho, Tableau). Eugene is a passionate speaker and SQL community volunteer presenting regularly at PASS SQL Saturday events and local user groups around Ukraine and Europe. Eugene is PASS Chapter Leader and he has a status MVP Data Platform https://www.linkedin.com/in/eugenepolonichko/ https://twitter.com/EvgenPolonichko
- 3. Agenda 1. What is Azure Databricks? • Azure Databricks • Apache Spark • Componets of Apache Spark • Architecture of Azure Databricks • Azure integration 2. Azure Databricks • Cluster • Workspace • Notebooks • Visualizations • Jobs and Alerts • Databricks File System • Business Intelligence Tools 3. For data engineer • Scenario • Prices
- 4. What is Azure Databricks?
- 5. Azure Databricks Azure Databricks is an Apache Spark- based analytics platform optimized for the Microsoft Azure cloud services platform. Designed with the founders of Apache Spark, Databricks is integrated with Azure to provide one-click setup, streamlined workflows, and an interactive workspace that enables collaboration between data scientists, data engineers, and business analysts.
- 6. Apache Spark-based analytics platform Azure Databricks comprises the complete open-source Apache Spark cluster technologies and capabilities. Spark in Azure Databricks includes the following components
- 7. Apache Spark-based analytics platform • Spark SQL and DataFrames: Spark SQL is the Spark module for working with structured data • Streaming: Real-time data processing and analysis for analytical and interactive applications. Integrates with HDFS, Flume, and Kafka. • MLib: Machine Learning library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as underlying optimization primitives. • GraphX: Graphs and graph computation for a broad scope of use cases from cognitive analytics to data exploration. • Spark Core API: Includes support for R, SQL, Python, Scala, and Java.
- 8. Architecture of Azure Databricks
- 9. Total Azure integration • Diversity of VM types • Security and Privacy • Flexibility in network topology • Azure Storage and Azure Data Lake integration • Azure Power BI • Azure Active Directory • Azure SQL Data Warehouse, Azure SQL DB, and Azure CosmosDB:
- 10. Azure Databricks
- 11. Clusters Azure Databricks clusters provide a unified platform for various use cases such as running production ETL pipelines, streaming analytics, ad-hoc analytics, and machine learning. Job Interactive
- 12. Workspace The Workspace is the special root folder for all of your organization’s Azure Databricks assets. The Workspace stores: • notebooks • libraries • dashboards • folders
- 13. Notebooks A notebook is a web-based interface to a document that contains runnable code, visualizations, and narrative text. • Create a notebook • Delete a notebook • Control access to a notebook • Notebook external formats • Notebooks and clusters • Schedule a notebook • Distributing notebooks
- 14. Visualizations Databricks supports a number of visualizations out of the box. All notebooks, regardless of their language, support Databricks visualization using the display function. display(<dataframe-name>)
- 15. Jobs and Alerts A job is a way of running a notebook or JAR either immediately or on a scheduled basis The number of jobs is limited to 1000.
- 16. Alerts You can set up email alerts for job runs. You can send alerts up job start, job success, and job failure (including skipped jobs), providing multiple comma-separated email addresses for each alert type. You can also opt out of alerts for skipped job runs.
- 17. Databricks File System Databricks File System (DBFS) is a distributed file system installed on Databricks Runtime clusters. Files in DBFS persist to Azure Blob storage You can access files in DBFS using the Databricks CLI, DBFS API, Databricks Utilities, Spark APIs, and local file APIs. # List files in DBFS dbfs ls # Put local file ./apple.txt to dbfs:/apple.txt dbfs cp ./apple.txt dbfs:/apple.txt # Get dbfs:/apple.txt and save to local file ./apple.txt dbfs cp dbfs:/apple.txt ./apple.txt # Recursively put local dir ./banana to dbfs:/banana dbfs cp -r ./banana dbfs:/banana Python Copy #write a file to DBFS using python i/o apis with open("/dbfs/tmp/test_dbfs.txt", 'w') as f: f.write("Apache Spark is awesome!n") f.write("End of example!") # read the file with open("/dbfs/tmp/test_dbfs.txt", "r") as f_read: for line in f_read: print line
- 18. Business Intelligence Tools Business Intelligence (BI) tools can connect to Azure Databricks clusters to query data in tables. Every Azure Databricks cluster runs a JDBC/ODBC server on the driver node. This section provides general instructions for connecting BI tools to Azure Databricks clusters, along with specific instructions for popular BI tools.
- 20. Scenario
- 21. Scenario
- 22. Thank you