Spark Streaming with Azure Databricks

Editor's Notes

• #2: In the world of data science we often default to processing in nightly or hourly batches, but that pattern is not enough any more. Our customers and business leaders see information is being created all the time and realize it should be available much sooner. While the move to stream processing adds complexity, the tools we have available make it achievable for teams of any size. In this session we will talk about why we need to shift some of our workloads from batch data jobs to streaming in real-time. We'll dive into how Spark Structured Streaming in Azure Databricks enables this along with streaming data systems such as Kafka and EventHub. We will discuss the concepts, how Azure Databricks enables stream processing, and review code examples on a sample data set.
• #4: Shifting to Streaming: We don’t have to convince our stakeholders that they don’t really need streaming. Understand the needs, find the right uses for streaming, and make it happen. Discuss pros and cons, considerations before going to production, and general use cases in AI/ML Spark, Event Hubs, and Kafka Define the systems we will be using for this session, including some of the reasons we choose them Talk about some of the options for using these together Getting Hands On Review dependencies that are not covered Walk through basic setup of the most important pieces Demo of use case code, highlight some important Structure Streaming components Best Practices Cover things to consider when working with Spark Structured Streaming and Kafka or Event Hubs
• #7: In the world of data science, those of us who develop ETL pipelines have determined that everything can be processed in nightly or hourly batches, but that only makes sense to data engineers. Our customers and business leaders see information is being created all the time and realize it should be available much sooner.
• #8: Dealing with a large set of data at once brings its own challenges (a lot of resources at once, large table joins, run out of memory, etc) Process as it comes in for cleaner logic (rather than seeing latest state, we see events as they happen and update state downstream) Even if not doing real-time analytics yet, prepare for when you will - the times they are a’changin
• #11: A fast and general engine for large-scale data processing, uses memory to provide benefit Often replaces MapReduce as parallel programming api on Hadoop, the way it handles data (RDDs) provides one performance benefit and use of memory when possible provides another large performance benefit Can run on Hadoop (using Yarn) but also as a separate Spark cluster. Local is possible as well but reduces the performance benefits…I find its still a useful API though Run Java, Scala, Python, or R. If you don’t already know one of those languages really well, I recommend trying it in Python and Scala and pick whichever is easiest for you. Several modules for different use cases, similar api so you can swap between modes relatively easily. For example, we have both streaming and batch sources of some data and we reuse the rest of the spark processing transformations.
• #12: A fast and general engine for large-scale data processing, uses memory to provide benefit Often replaces MapReduce as parallel programming api on Hadoop, the way it handles data (RDDs) provides one performance benefit and use of memory when possible provides another large performance benefit Can run on Hadoop (using Yarn) but also as a separate Spark cluster. Local is possible as well but reduces the performance benefits…I find its still a useful API though Run Java, Scala, Python, or R. If you don’t already know one of those languages really well, I recommend trying it in Python and Scala and pick whichever is easiest for you. Several modules for different use cases, similar api so you can swap between modes relatively easily. For example, we have both streaming and batch sources of some data and we reuse the rest of the spark processing transformations.
• #13: A fast and general engine for large-scale data processing, uses memory to provide benefit Often replaces MapReduce as parallel programming api on Hadoop, the way it handles data (RDDs) provides one performance benefit and use of memory when possible provides another large performance benefit Can run on Hadoop (using Yarn) but also as a separate Spark cluster. Local is possible as well but reduces the performance benefits…I find its still a useful API though Run Java, Scala, Python, or R. If you don’t already know one of those languages really well, I recommend trying it in Python and Scala and pick whichever is easiest for you. Several modules for different use cases, similar api so you can swap between modes relatively easily. For example, we have both streaming and batch sources of some data and we reuse the rest of the spark processing transformations.
• #16: Window is essentially like grouping. Continuously compute the average distance for each vendor over the last 10 minutes
• #17: Window is essentially like grouping. Continuously compute the average distance for each vendor over the last 10 minutes
• #30: Quick overview of important databricks workspace segments – Clusters, Tables, Notebooks Open create_parquet_tables notebook and run first few commands as examples of working without delta