Understanding data resiliency

- [Instructor] Let's start with understanding the basics of data resiliency. My goal is to not only explain what is data resiliency, but also why is it important with practical use cases related to moving data between systems and applications. Dealing with money or financial information is always a great use case to explain the importance of data resiliency. Imagine you have an online bank account, it's payday. You log in expecting to see the results of your hard work. You've been putting in overtime just to make the ends meet. But what if a technical issue occurs during the processing? Your job sends the payment to your bank, but for some reason, the payment was not successfully delivered, or worse, it was totally lost. You check your balance expecting to be well compensated, assuming the data will be correct based on all your hard work, but something is terribly wrong. I remember my mother working two jobs to provide for our family. Back then, payment processing was all paper-based. Nowadays, everything is electronic. Many people depend on the reliable delivery of information like electronic payments. So we need data resiliency. And data resiliency is the ability for data to withstand or recover quickly from technical issues such as networking issues, process failures, or even hardware failures like a disc failure, or even human accidents, or natural disasters such as hurricanes and earthquakes. With resiliency, systems are designed to overcome these technical challenges so we all can just enjoy our lives. Here is an architectural review of an e-payment and account balance app. The payment is published from the payment app to the account balance app. The account balance app will process the payments and the results are stored in some sort of storage disc, typically a database. But how do you design the system such that the payments are delivered, even if there's an issue with the account balance app or with the database itself? Let's go through a couple of examples. One solution is to have a second copy of the application that's ready to receive those payments when the primary goes down. But what about the state of the account balance in the secondary? How do you ensure it is correct? And what if it goes down due to some sort of networking issue? In this course, I'll introduce you to a broker. The goal of the broker is to facilitate reliable delivery of payments. It handles sending payments to the primary and backup apps if there's any issues with the primary. It can hold payments. So for example, if the account balance app is not ready to receive the data, that data can be delivered at a later point in time. You can use clustering, which is basically multiple brokers that cover for each other if one has an issue. The cluster can overcome partial failures. The loss of a single broker can be transparent to the payment and the account balance app. I will show you one of the most popular brokers for reliable message delivery and it's called RabbitMQ. So let's begin.