From the course: Complete Guide to Databricks for Data Engineering

Unlock this course with a free trial

Join today to access over 24,700 courses taught by industry experts.

Handle timestamp manipulation in PySpark

Handle timestamp manipulation in PySpark - Databricks Tutorial

From the course: Complete Guide to Databricks for Data Engineering

Start my 1-month free trial Buy for my team

Handle timestamp manipulation in PySpark

“

- [Instructor] Date is one part, but definitely when you work in a real projects as a data engineer, you will get the timestamp column as well. If you look at it our current data frame, you will find that it also has one column that is of a timestamp type. That is the ingestion timestamp. So how we can play with this timestamp column? Let's just see that. Very similar to the way we were using the date column. In the timestamp column, also, you have lots of those functions out there. For example, if from the timestamp, if you want to extract some specific information, we can do that. We say select here, ingestion timestamp, and then let's say I want to extract a year out of that. I can do year and then I give the ingestion timestamp column name then month, then I can also get day of the month. All these things we were getting it from the date, as well. However, in the case of a timestamp, we have some extra columns, like hours. We want to get the specific hour from that minute or the…

Contents