From the course: End-to-End Real-World Data Engineering Project with Snowflake

Snowflake project architecture diagram - Snowflake Tutorial

From the course: End-to-End Real-World Data Engineering Project with Snowflake

Start my 1-month free trial Buy for my team

Snowflake project architecture diagram

“

- [Instructor] Let's look at the high level architecture diagram of this project. You can see that the external system, like CRM, inventory, and transaction, the data is coming from all these three system to our storage account. That is Azure Data Lake Storage. And once the data comes in at Azure Data Lake Storage account, our work starts from there. It may be based on their frequency, like on a daily basis or an hourly basis. These systems will set the data into storage account. And the best part of the storage account is it can store any kind of a data. Now the data from the storage account will move to the Snowflake bronze layer schema. Now in the snowflake are one database we're going to create, at this one database will have three separate schema. I have showcased it using the database itself, but there are no three different database, there is just one database only, and one database will have three separate schema for bronze, silver, and gold. And the data will move in between them from bronze to silver, silver to gold. And once our gold layer data get created and available, this could be fed into the reporting tools. That's how the architecture of this project looks like. The major key components of this project, which we're going to build probably, is Snowflake database. We're going to create the external stage to move the data from ADLS to bronze. We need to create the multiple schema, bronze, silver, gold. We need to create the tables in all the layers. We need to create the task, which will automatically pull the data from ADLS to bronze and from bronze to silver. We also have a stream get created so that we can get the incremental data from bronze to silver. And lastly, we'll have views in our gold layer schema. Great, so we got a plan. What are we are going to do? Let's start implementing.

Contents