Implementation behind Databricks Delta Live Tables(DLT), using python decorators
Delta Live Tables (DLT) is a declarative ETL framework for the Databricks Data Intelligence Platform that helps data teams simplify streaming and batch ETL cost-effectively.
For Python developers, Delta Live Tables has a Python interface designed to support the loading and transformation of data. For tasks that require processing not supported by SQL, developers can use Python to write pipeline source code that combines Delta Live Tables queries with Python functions that implement the processing not supported by the Delta Live Tables interfaces.
DLT uses declarative frame work based on python decorators This article is is not going to explain about pipelines of DLT, instead explaining the basics of python decorators which is essential to know before building any DLT pipelines
In Python, a decorator is a function that allows you to modify the behavior of another function without changing its core logic. It takes another function as an argument and returns the function with extended functionality. This way, you can use decorators to add some extra logic to existing functions to increase reusability with just a few lines of code. In this article, we will explore eight built-in Python decorators that can help you write more elegant and maintainable code.
Decorator Chaining : Multiple decorators can be chained in Python.
To chain decorators in Python, we can apply multiple decorators to a single function by placing them one after the other, with the most inner decorator being applied first.