Bringing structure to agentive AI data

- How do we structure our data in such a way that when there is some sort of conflict within the data, the data systems, these AI agents, are able to identify these conflicts and then resolve them? - If you think of this, this is the scope of your project, 60 to 80% of the project was just getting the right data, preparing the training datasets, and doing all of that. We live in a new world here today of large language models, small language models, and narrow AI models, where it's coming pre-trained. It needs fine-tuning. Sometimes it needs fine-tuning, sometimes it needs embedding, sometimes it needs vector indexes to tune it to the the relevance factor that you want to keep in mind. But at the base level, if I step back and look at the bigger picture, it comes back to the basics of ensuring that you have a process to clean the data, bring it all in, transform it, put it in a shape that is usable, and then use that for building out your model or building out the application, which is relying on a pre-trained model. But at the same time, you need to think about feature selection. What are the right parameters? Sometimes there's too many parameters. You need to focus on certain selection techniques for selecting the right features, to focus on the right patterns, and then get the right results out of it. And then as part of this, you have to also think about the regular rhythm of the business, per se, for a data engineer or data analyst. You need to do data audits regularly, you need to have the data governance. AI is only as good as the data that is fed into it, so we'll end up with a lot of good quality data sometimes, but still not usable, and that's where we need to think about it more from a process angle, the process that is important in that stage. So it could be around the data validation, it could be around access controls, the CFO saying, "I have financial data, I want to run an AI app on top of it, but don't share this with someone else." So you have to think about audit trails, you have to think about data encryptions. There are so many tools out there, if I show you the picture today for the data and AI landscape, so many vendors, so many solutions, it just fills up the whole, it's really an eye chart if you look at it. So we have to think about this more from a process-centric approach in that perspective. You have certain human capabilities, you have to keep human in the loop all the while as you're working through this AI build out, but you have to think about how the data gets there. Is the data in motion? Is the data at rest? Is it secure? Is it available? All of those aspects come into play here, and also training people on that, because employees in the organization need to know how to use the data for AI, and sometimes, when you make it too generic, people are dissatisfied, because it doesn't cater to their needs. You need the people, process, and tooling to ensure data integrity and ensure that the data is usable by the AI agents that you're building out. And there are many, many tools out there that you can go through and think about. You have to think about also the ecosystem of tool vendors who are available to you and which you can apply in your specific industry.