From the course: Complete Guide to Databricks for Data Engineering
Join today to access over 24,700 courses taught by industry experts.
Write a DataFrame as using partitioning - Databricks Tutorial
From the course: Complete Guide to Databricks for Data Engineering
Write a DataFrame as using partitioning
- [Instructor] At the time of writing, you can define the partitions. How? Let's just see. I can say that df.write.mode, let's say, I'm giving in mode as an overwrite mode and I can say option. If I'm writing it as a CSV file, I can say that, okay, let's keep header equal to true. And then, if I wanted to do in a partition manner, I can just say partition by. So, partition by, you need to define the column, on which basis you want to partition it. For example, I want to partition it based on that customer type. You remember? We had three customer type: VIP, premium, and regular. So I wanted to partition this data frame on these three different datatypes and then I can just simply say, .csv, and I can give the path and I'll keep the path same, filestore, tables, output, CSV, partition. So I give a new path like CSV partition. And let me execute this. The only change which we have done this time is, we have added this partition by. Now, see the magic. To see the magic, let's go to our…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.