Spark Structured Streaming
2 likes276 views
The document discusses Spark Structured Streaming, comparing different data abstractions like RDDs, DataFrames, and Datasets, and their respective advantages and disadvantages. It highlights the programming model, the need for defining schemas, and various output modes for streaming applications. Additionally, it outlines operations one can perform on streaming DataFrames and Datasets, emphasizing the importance of type safety and optimization.
Spark Structured Streaming
- 2. Agenda ● Streaming – What and Why ? ● Rdd vs DataFrames vs Datasets ● Programming Model ● Streaming DataFrames and Datasets ● Defining Schema ● Output Modes ● Basic operations ● Window operations on event time*
- 3. RDD Some key features of RDD :- ● Resilient ● Type Safe ● Immutable ● Lazy evaluation and many more
- 4. Problems with RDDs ● They express the ‘how’ of a solution better than the ‘what’. Rdd library is bit opaque. ● They cannot be optimized by Spark. ● It’s too easy to build an inefficient RDD transformation chain.
- 5. DataFrames ● DataFrame API provides a higher-level abstraction, allowing you to use a query language to manipulate data. ● Avail SQL functionalities. ● Focus on What rather than on How.
- 6. DataFrames “Let spark figure out how to do it for you. Like, in RDBMS we just fire the sql queries and are not concerned how the query brings out the data, i.e., we just care about the result not the process”
- 7. DataFrames ● There are three types of logical plans:- ● Parsed logical plan :- Checks column names, table names, etc. ● Analyzed logical plan :- Analysis of the query. ● Optimized logical plan :- Perform optimizations. ● Physical plan(Actual RDD)
- 9. Problems with DataFrames ● Till now, everything looks cool, But?? ● We have lost type safety ●
- 10. Solution ? ● We can convert dataframes back into rdds, but then we loose upon the optimization. ● We would like to get back our compile time safety without giving up the optimizations.
- 11. Datasets ● Extension to the DataFrame API. ● Conceptually similar to RDDs. (You can actually operate on objects) ● Interoperate more easily with dataframe api. ● Like Rdd, Dataset has a type. ● They use Encoders for serialization/deserialization which are quite fast compared to java/kryo.
- 12. Datasets ● Providing type to previous example in dataset
- 15. Output Modes ● The “Output” is defined as what gets written out to the external storage. The output can be defined in a different mode: ● Complete Mode - The entire updated Result Table will be written to the external storage. ● Append Mode - Only the new rows appended in the Result Table since the last trigger will be written to the external storage. This is applicable only on the queries where existing rows in the Result Table are not expected to change. ● Update Mode - Only the rows that were updated in the Result Table since the last trigger will be written to the external storage. Note that this is different from the Complete Mode in that this mode only outputs the rows that have changed since the last trigger. If the query doesn’t contain aggregations, it will be equivalent to Append mode.
- 16. Streaming DataFrames and Datasets ● Streaming DataFrames can be created through the DataStreamReader interface returned by SparkSession.readStream(). ● For example, defining streaming dataframe from kafka (data source) ●
- 17. Defining Schema ● Structured streaming requires you to specify the schema. There are two ways you can specify schema:- 1. You could build your schema manually
- 18. Defining Schema 2. Use the business object that describes the dataset
- 19. Operations on streaming DataFrames/Datasets ● You can apply various kind of operations on streaming DataFrames/Datasets. ● Some of the basic operations are selection, projection, aggregation, etc.
- 20. Window Operations on Event time
- 21. References ● https://spark.apache.org/docs/latest/structured-stre ● https://www.youtube.com/watch?v=pZQsDloGB4w& ● https://jaceklaskowski.gitbooks.io/spark-structured-
- 22. Thank you