Opening, 1st International Workshop on Patterns and Practices of Reliable AI Engineering and Governance (AI-Pattern'24)
0 likes90 views
The 1st International Workshop on Patterns and Practices of Reliable AI Engineering and Governance (AI-Pattern'24) aims to enhance understanding and collaboration among researchers and practitioners in AI engineering and governance, focusing on theoretical, social, and practical advancements. It will feature discussions on effective AI/ML software engineering patterns, including architecture, assurance arguments, and responsible design patterns, to solve common challenges in the field. The workshop will include sessions on extracting learning patterns, cybersecurity assessments, and approaches for engineering safe and responsible AI systems.
![AI/ML software engineering patterns
• Architecture and design patterns
– Software Engineering Patterns for ML
applications [SEP4MLA]
– Machine Learning Design Patterns
[MLDP]
• Assurance argument patterns
– Safety Case Pattern for ML systems
[Safety]
– Security Argument Patterns for DNN
[Security][Security2]
• Responsible AI patterns
– Responsible AI System Design Patterns
[Responsible]
• Development and management
practices
– Lifecycle phase practices [Practice1]
– Issues and development practices
[Practice2]
• Prompt engineering patterns
– Prompt Pattern Catalog and Taxonomy
[Prompt][Prompt2]
7
[MLDP] V. Lakshmanan, et al., “Machine Learning Design Patterns,” O’Reilly, 2020
[SEP4MLA] H. Washizaki, et al. “Software Engineering Design Patterns for Machine Learning Applications,” IEEE Computer 55(3) 2022
[Safety] E. Wozniak, et al., “A Safety Case Pattern for Systems with Machine Learning Components,” SAFECOMP 2020 Workshop
[Security] M. Zeroual, et al., “Security Argument patterns for Deep Neural Network Development,” PLoP 2023
[Security2] M. Mutsche, et al. “Robustness-based Security Case Verification for Deep Neural Networks,” AsianPLoP 2024
[Responsible] Q. Lu, et al., “Responsible-AI-by-Design: a Pattern Collection for Designing Responsible AI Systems,” IEEE Software, 2023
[Practice1] M. S. Rahman, et al., “Machine Learning Application Development: Practitioners’ Insights,” Software Quality Journal, 31, 2023
[Practice2] Y. Watanabe, et al., “Preliminary Literature Review of Machine Learning System Development Practices,” COMPSAC 2021 Fast Abstract
[Prompt] J, White, et al., “A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT,” arXiv 2302.11382, 2023
[Prompt2] Y. Sasaki, et al., “A Taxonomy and Review of Prompt Engineering Patterns in Software Engineering,” COMPSAC 2024](https://image.slidesharecdn.com/ai-pattern24-openiing-241104010134-70a64982/85/Opening-1st-International-Workshop-on-Patterns-and-Practices-of-Reliable-AI-Engineering-and-Governance-AI-Pattern-24-7-320.jpg)
Opening, 1st International Workshop on Patterns and Practices of Reliable AI Engineering and Governance (AI-Pattern'24)
- 1. 1st International Workshop on Patterns and Practices of Reliable AI Engineering and Governance (AI-Pattern'24) Hironori Washizaki (Waseda University), Nobukazu Yoshioka (QAML Inc), Naoyasu Ubayashi (Waseda University), Emiliano Tramontana (Università di Catania) October 28th, 2024, in Tsukuba, Japan
- 2. Thanks to program committee • Shaukat Ali (Simula Research Laboratory) • Qinghua Lu (Data61, CSIRO) • Foutse Khomh (Polytechnique Montreal) • Hironori Takeuchi (Musashi University) • Ademar Aguiar (Universidade do Porto) • Eduardo Guerra (Free University of Bolzen-Bolzano) • Joseph Yoder (The Refactory) • Raja Rao Budaraju (Oracle) • Yu-Chin Cheng (National Taipei University of Technology) • Kyle Brown (IBM) • Shinpei Hayashi (Tokyo Institute of Technology) • Takao Okubo (Institute of Information Security) • Shinpei Ogata (Shinshu University) • Yann-Gaël Guéhéneuc (Concordia University) 2
- 3. Goal of the workshop • Seeks to improve understanding of the theoretical, social, technological, and practical advances and issues related to patterns and practices in reliable AI engineering and governance. • Provides the opportunity to bring together researchers and practitioners and discuss the future prospects of this area. 3
- 4. Street Cafe Problem: Needs to have a place where people can sit lazily, legitimately, be on view, and watch the world go by… Solution: Encourage local cafes to spring up in each neighborhood. Make them intimate places, with several rooms, open to a busy path … Alexander, Christopher, et al. A Pattern Language. Oxford University Press, 1977. 4 https://tokyocheapo.com/food-and-drink/drinking/tokyo-terrace-tippling/
- 5. Example case of ML-based system design • We wish to identify the type of instrument for the sound picked up by the phone and achieve recording and response according to the type. • However, the memory and performance of the phone is limited, and a large deep learning model is unlikely to be loaded. How can we do this? 5 Pretrained Model • Let's use Two-stage predictions where a small model on the phone determines if a sound is a musical instrument, and a large model on the cloud classifies the type of sound only if it is a musical instrument. • For the large model, we will adopt Transfer Learning to achieve precise classification. Machine Learning Design Patterns (V. Lakshmanan, et al. 2020) Two-stage predictions • Problem: There is a need to maintain the performance of models that are large and complex in nature, even when deployed to edge or distributed devices. • Solution: The utilization flow is divided into two phases, with only the simple phase performed on the edge. Transfer Learning • Problem: There is a lack of large data sets needed to train complex machine learning models. • Solution: Some layers of the trained model are taken out and the weights are frozen and used in the new model to solve similar problems without being trained.
- 6. AI/ML software engineering needs patterns! • Bridge between abstract paradigms and concrete cases/tools – Documenting Know-Why, Know-What and Know-How – Reusing solutions and problems – Getting consistent architecture • Common language among stakeholders – Software engineers, data scientist, domain experts, network engineers, … 6 Paradigm Case Tool FW Instruction ？ ？
- 7. AI/ML software engineering patterns • Architecture and design patterns – Software Engineering Patterns for ML applications [SEP4MLA] – Machine Learning Design Patterns [MLDP] • Assurance argument patterns – Safety Case Pattern for ML systems [Safety] – Security Argument Patterns for DNN [Security][Security2] • Responsible AI patterns – Responsible AI System Design Patterns [Responsible] • Development and management practices – Lifecycle phase practices [Practice1] – Issues and development practices [Practice2] • Prompt engineering patterns – Prompt Pattern Catalog and Taxonomy [Prompt][Prompt2] 7 [MLDP] V. Lakshmanan, et al., “Machine Learning Design Patterns,” O’Reilly, 2020 [SEP4MLA] H. Washizaki, et al. “Software Engineering Design Patterns for Machine Learning Applications,” IEEE Computer 55(3) 2022 [Safety] E. Wozniak, et al., “A Safety Case Pattern for Systems with Machine Learning Components,” SAFECOMP 2020 Workshop [Security] M. Zeroual, et al., “Security Argument patterns for Deep Neural Network Development,” PLoP 2023 [Security2] M. Mutsche, et al. “Robustness-based Security Case Verification for Deep Neural Networks,” AsianPLoP 2024 [Responsible] Q. Lu, et al., “Responsible-AI-by-Design: a Pattern Collection for Designing Responsible AI Systems,” IEEE Software, 2023 [Practice1] M. S. Rahman, et al., “Machine Learning Application Development: Practitioners’ Insights,” Software Quality Journal, 31, 2023 [Practice2] Y. Watanabe, et al., “Preliminary Literature Review of Machine Learning System Development Practices,” COMPSAC 2021 Fast Abstract [Prompt] J, White, et al., “A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT,” arXiv 2302.11382, 2023 [Prompt2] Y. Sasaki, et al., “A Taxonomy and Review of Prompt Engineering Patterns in Software Engineering,” COMPSAC 2024
- 8. AI architecture patterns IoT design patterns Towards pattern languages across layers and properties … OK, so, to attract many people to our city, Small Public Squares should be located in the center. At the SMALL PUBLIC SQUARE, make Street Cafes be Opening to the Street ... 8 https://unsplash.com/photos/EdpbTj3Br-Y https://unsplash.com/photos/GqurqYbj7aU https://unsplash.com/photos/zFoRwZirFvY AI management practices AI assurance argument patterns Governance Responsible AI patterns AI design patterns Small Public Square Street Cafe Opening to the Street
- 9. • Problem: … (AI/ML) pattern engineering • Extraction: Identifying and formulating recurring problems and solutions into “new” patterns to have reusable patterns • Detection: Detecting “known” patterns in software processes and products to comprehend and identify further improvement opportunities • Application: Selecting, concretizing and deploying patterns on software processes and products to resolve particular problems • Organization: Organizing patterns to build a system (i.e., language) of patterns • Integration: Integrating into pattern-oriented development and management 9 • Problem: … • Solution: …. AI/ML pattern Extraction Application Similar results Detection Pattern instances Organization Process Management Integration
- 10. Program • 9:00-10:30 Session I – Opening – Invited talk: A Pattern-Oriented Approach for Engineering Safe and Responsible AI Systems, Qinghua Lu (Data61, CSIRO) – Toward Pattern-Oriented Machine Learning Reliability Argumentation, Takumi Ayukawa, Jati H. Husen, Nobukazu Yoshioka, Hironori Washizaki and Naoyasu Ubayashi (Waseda University) • 11:00-12:30 Session II – A Process Pattern for Cybersecurity Assessment Automation: Experience and Futures, James Cusick (Ritsumeikan University) – Toward Extracting Learning Pattern: A Comparative Study of GPT-4o-mini and BERT Models in Predicting CVSS Base Vectors, Sho Isogai, Shinpei Ogata (Shinshu University), Yutaro Kashiwa (Nara Institute of Science and Technology), Satoshi Yazawa (Voice Research, Inc.), Kozo Okano (Shinshu University), Takao Okubo (Institute of Information Security), Hironori Washizaki (Waseda University) – Discussion – Closing Remarks 10
- 11. 11
- 12. 12