The data lifecycle of agentive AI

- So let's talk about the lifecycle here, the data lifecycle. Because if you're going to do this, we need to figure out how to make it flow. - Yeah. - Get the data through the entire machine and manage it all. What does that look like at a practical level? - Yes, practically speaking, every company today is a data company. Think about it, data is the fuel, which empowers everybody to work it. It's the fuel for AI too. And so the data engineers who exist in your organization today already know where the data is. Maybe it's not all cataloged, maybe it's not governed data. The first step would be to do a catalog or inventory of all these assets, wherever they are. They could be on Amazon, S3, they could be in Google Cloud, it could be in Microsoft Dataverse, or Azure. Wherever it is, that catalog of assets need to be pulled together so that you can do a data assessment and create that data landing zone that I was talking about. So think about unifying those, having a strategy for bringing them together, building the shortcuts or mirrored connections or make it, you know, into one lake that you can use effectively for all of your activity. Now, there are two types of activities, typically, three actually, if you think about it. You have the citizen user, you know, the super user who just says, I don't care about all of the backend processes. All I need is to make a decision. Give me the right chart and the right metrics at the right point, and I'll make the decision. So this super user is more like a Power BI, or you know, a visual user who's looking at a chart and making a decision. So they are one class of users. But then the second class are people who are using data warehouses because they're doing analytics on past data, historical data. So the one lake also includes a piece which is related to a warehouse. That's the lake house aspect of it, right? The warehouse that you're going to use for all of your analytics. So you're to build that portion out because you have loads and loads of structured and unstructured data available today. Maybe start with structured first because it's the easier piece. But with fabric for per se, you can have a hybrid data mesh architecture and bring unstructured data also in there. When I say unstructured, it means PDF documents and all the other kind of, you know, unstructured pictures, images, videos, audio. Everything can be pulled in into this one approach that you have. So now you have the data, you have a warehouse also. The next thing to keep in mind is you can have a bunch of processes to maintain the pipeline of data flowing in, make sure it's fresh, it's available. But the next thing to consider is what is the real-time component of this? What do you need in real-time to make the right decisions? Maybe you have an area to focus on, which you need to improve upon. So you've already, you have the warehouse, you've transformed it, you have enriched it. It's available to people in your organization. Even the data scientists can use it. But then there's this real-time aspect of data coming in every second, every minute into the organization. Depends on the type of business, obviously, but then you have to make that data available. That data has to be observable, it has to be governed too. It has to be safe and secure, and all of it comes together. And now truly, you have a holistic and comprehensive view on which you can put semantic layers and build a business intelligence layer that you need to make the decisions. Now, some portions of this data scientists and AI engineers will use as raw data, but high quality data, good quality data to train their models. Some of it'll be used for pre-trained models where you are grounding it in data and saying, I need the most recent sales numbers from this X, Y, Z customer to project the growth of my company and organization. So that needs to be grounded in current data, real data that's available. And some of it'll be real-time data where you're to, you're sitting in the store or you're on the point of sale terminal or a mobile application and you're doing a transaction and you need to make a recommendation or a prediction on the offer that needs to be made to the customer or consumer so that they can make the right decision. So all these types of users are there, and that's how everything comes together in the form of a data foundation on which you put your AI orchestration, on which you put your AI agents or applications, on which you have the user experiences all crafted in a beautiful way so that the end user or the super user or the data scientists or AI engineer can make the right decisions and have a good construct in place to get to the eventual result.