ICAART 2025 presentation on Stateful Monitoring and Responsible Deployment of AI Agents
1 like234 views
The document discusses the evolution and complexities of agentic AI, highlighting task decomposition into simpler tasks executed by agents, both dynamically and statically. It addresses challenges in observability and stateful execution for AI agents, emphasizing the importance of monitoring their performance due to inherent non-determinism and privacy constraints. Additionally, it touches on responsible deployment, data quality issues, explainability, and privacy risks associated with AI and LLMs.
ICAART 2025 presentation on Stateful Monitoring and Responsible Deployment of AI Agents
- 1. Stateful Monitoring and Responsible Deployment of AI Agents Debmalya Biswas Wipro AI
- 3. (Complex) Agentic AI Task Decomposition A high-level approach to solving complex tasks: • - decomposition of the given complex task into a hierarchy or workflow of) simple tasks, followed by • - composition of agents able to execute the simpler tasks. This can be achieved in a dynamic or static manner. • Dynamic: given a complex user task, the system comes up with a plan to fulfil the request depending on the capabilities of available agents at run-time. • Static: given a set of agents, composite agents are defined manually at design-time combining their capabilities. * D. Biswas. Constraints Enabled Autonomous Agent Marketplace: Discovery and Matchmaking. 16th International Conference on Agents and Artificial Intelligence (ICAART), 2024 (link)
- 4. Agentic AI Platform Reference Architecture The future where enterprises will be able to develop new Enterprise AI Apps by orchestrating / composing multiple existing AI Agents.
- 6. Non-determinism in Agentic AI Systems There are two non-deterministic operators in the execution plan: ‘Check Credit’ and ‘Delivery Mode’. The choice ‘Delivery Mode’ indicates that the user can either pick-up the order directly from the store or have it shipped to his address. Given this, shipping is a non- deterministic choice and may not be invoked during the actual execution.
- 7. Observability Challenges for Agentic AI Observability for AI Agents is challenging: - No global observer: Due to their distributed nature, we cannot assume the existence of an entity having visibility over the entire execution. In fact, due to their privacy and autonomy requirements, even the composite agent may not have visibility over the internal processing of its component agents. - Parallelism: AI agents allow parallel composition of processes. - Dynamic configuration: The agents are selected incrementally as the execution progresses (dynamic binding). Thus, the “components” of the distributed system may not be known in advance.
- 8. Stateful execution for AI Agents AgentOps monitoring is critical given the complexity and long running nature of AI agents. We define observability as the ability to find out where in the process the execution is and whether any unanticipated glitches have appeared. - Local queries: Queries which can be answered based on the local state information of an agent. - Composite queries: Queries expressed over the states of several agents. - Historical queries: Queries related to the execution history of the composition. - Relationship queries: Queries based on the relationship between states.
- 10. Data Quality Issues with respect to LLMs, esp. Vector DBs From a data quality point of view, we see the following challenges w.r.t. LLMs, esp. Vector DBs: - Accuracy of the encodings in vector stores, measures in terms of correctness and groundedness of the generated LLM responses. - Incorrect and/or inconsistent vectors: Due to issues in the embedding process, some vectors may end up getting corrupted, be incomplete, or getting generated with a different dimensionality. - Missing data can be in the form of missing vectors or metadata. - Timeliness issues w.r.t. outdated documents impacting the vector store. * D. Biswas. Long-term Memory for AI Agents. AI Advances, 2024 (link) * D. Biswas. Long-term Memory for AI Agents. AI Advances, 2024 (link)
- 11. Explainability Explainable AI is an umbrella term for a range of tools, algorithms and methods; which accompany AI model predictions with explanations. - Explainability of AI models ranks high among the list of ‘non- functional’ AI features to be considered by enterprises. - For example, this implies having to explain why an ML model profiled a user to be in a specific segment — which led him/her to receiving an advertisement. (Labeled) Data Train ML Model Predictions Explanation Model Explainable Predictions
- 12. Fairness & Bias Bias creeps into AI models, primarily due to the inherent bias already present in the training data. So the ‘data’ part of AI model development is key to addressing bias. - Historical Bias: arises due to historical inequality of human decisions captured in the training data - Representation Bias: arises due to training data that is not representative of the actual population. *H. Suresh, J. V. Guttag. A Framework for Understanding Unintended Consequences of Machine Learning, 2020 (link) *H. Suresh, J. V. Guttag. A Framework for Understanding Unintended Consequences of Machine Learning, 2020 (link)
- 13. ML Privacy Risks Two broad categories of privacy inference attacks: • Membership inference (if a specific user data item was present in the training dataset) and • Property inference (reconstruct properties of a participant’s dataset) attacks. Black box attacks are still possible when the attacker only has access to the APIs: invoke the model and observe the relationships between inputs and outputs. Training dataset wants access to ML Model (Classification, Prediction) Inference API has access to Attacker * D. Biswas. Privacy Preserving Chatbot Conversations. IEEE AIKE 2020: 179-182 (link) *D. Biswas, K. Vidyasankar. A Privacy Framework for Hierarchical Federated Learning. CIKM Workshops 2021 (link)
- 14. Gen AI Privacy Risks – novel challenges From a privacy point of view, we need to consider the following additional / different LLM privacy risks: - Membership and property leakage from pre-training data - Model features leakage from pre-trained LLM - Privacy leakage from conversations (history) with LLMs - Compliance with privacy intent of users * D. Biswas. Privacy Risks of Large Language Models. AI Advances, 2024 (link) * D. Biswas. Privacy Risks of Large Language Models. AI Advances, 2024 (link)
- 15. Responsible deployment of AI Agents * D. Biswas. Stateful Monitoring and Responsible Deployment of AI Agents. 17th International Conference on Agents and Artificial Intelligence (ICAART), 2025 (link)