Master tester AI toolbox - Kari Kakkonen at Testaus ja AI 2025 Professio
0 likes135 views
My slides at Professio Testaus ja AI 2025 seminar in Espoo, Finland. Deck in English, even though I talked in Finnish this time, in addition to chairing the event. I discuss the different motivations for testing to use AI tools to help in testing, and give several examples in each categories, some open source, some commercial.
Master tester AI toolbox - Kari Kakkonen at Testaus ja AI 2025 Professio
- 1. Copyright Dragons Out Oy and Gofore 2025 • Kari Kakkonen • https://www.linkedin.com/in/karikakkonen • Gofore • Professio Testaus ja AI • Espoo • 27.5.2025 1 Testauksen työkalupakki haltuun - Mastering the Testing Toolbox
- 2. ROLES • Gofore, Service Owner, Customer Expertise Development • Dragons Out Oy, CEO, Children’s and testing author • TMMi, Secretary, Board of Directors • Finnish Software Testing Board (FiSTB), Treasurer ACHIEVEMENTS • Tester of the Year in Finland 2021 • EuroSTAR Testing Excellence Award 2021 • Exemplary DevOps Instructor Award 2023 by DASA • ISTQB Executive Committee 2015-2021 • Influencing testing since 1996 • Ranked in 100 most influential IT persons in Finland (Tivi magazine) • Great number of presentations in Finnish and international conferences • TestausOSY/FAST founding member. • Author of Dragons Out testing book • Co-author of ACT2LEAD software testing leadership handbook • Co-author of Agile Testing Foundations book • Regular blogger in Tivi-magazine Kari Kakkonen Service Owner, Customer Expertise Development SERVICES • ISTQB Advanced, Foundation, Agile Testing, AI Testing • Quality Professional • DASA DevOps, Agile, Scrum, Product Management • Quality & Test process and organization development, Metrics, TMMi and other assessments • Agile testing, Scrum, Kanban, Lean • Leadership • Test automation, Mobile, Cloud, DevOps, AI • Quality, cost, benefits EDUCATION • ISTQB Expert Level Test Management & Advanced Full & Agile Tester certified • DASA DevOps, Scrum Master and SAFe certified • TMMi Professional, Assessor, Process Improver certified • SPICE provisionary assessor certified • M.Sc.(Eng), Helsinki University of Technology (present Aalto University), Otaniemi, Espoo • Marketing studies, University of Wisconsin-Madison, the USA. BUSINESS DOMAINS Wide spread of business domain knowledge: Embedded, industry, public, training, telecommunications, commerce, Insurance, banking, pension. 2 www.gofore.com www.dragonsout.com www.act2lead.net MORE INFORMATION linkedin.com/in/karikakkonen/ Copyright Dragons Out Oy and Gofore 2025
- 4. The book project ”Dragons Out!” • Mission • “Software testing brought to children” • Book • Author Kari Kakkonen • Illustrator Adrienn Széll • Text and illustration rights Dragons Out Oy • In Finnish, English, Polish, French and growing • For ages of 10-99 • Free “Dragon lesson in software testing” presentation under Creative Commons – license • Translated to 20 languages! • More info: www.dragonsout.com 4 Copyright Dragons Out Oy and Gofore 2025
- 5. ACT 2 LEAD software testing leadership handbook ● Easy to read - chapters can be read in any order. ● Structure: questions, answers and cases. ○ 34 main chapters = questions, see next page. ● 270+ pages, ○ in Finnish (softback, e-book) ○ in English (e-book). ● For people like CxO, director, head of, manager, product owner, designer, developer, test manager, tester and student. ● Teaches to lead testing, not to test. Buy the book: https://leanpub.com/act2lead Book website: www.act2lead.net 5
- 6. ISTQB GLOBAL PRESENCE • Number of exams administered: over 1,4 million • Number of certifications issued: over 1 million • In 130 countries Copyright Dragons Out Oy and Gofore 2025 6
- 7. TMMi for test improvement in all kinds of testing, including Agile, DevOps, AI Copyright Dragons Out Oy and Gofore 2025 7
- 8. • Mitä työkaluja tällä hetkellä on käytössä? • Mitkä ovat Gen AI:n vahvuudet tällä hetkellä? • Käytännön esimerkkejä parhaista työkaluista • What tools are currently in use? • What are the strengths of Gen AI at the moment? • Practical examples of the best tools. Agenda Copyright Dragons Out Oy and Gofore 2025 8
- 9. AI augmented testing • Gartner term • https://www.gartner.com/en/documents/5194063 • “Software engineering leaders are now prioritizing development productivity to enhance market responsiveness and build software applications more efficiently, while also aiming to maintain high quality. To meet this challenge they are increasingly turning to AI- augmented testing tools.” Copyright Dragons Out Oy and Gofore 2025 9
- 11. Anomaly detection Copyright Dragons Out Oy and Gofore 2025 11
- 12. • Can I find defects faster from lots of test runs? • Which automatic defect reports / crash reports are actual defects? • Are there duplicate reports? • Automation to defect reporting? Tester’s objectives Copyright Dragons Out Oy and Gofore 2025 12
- 13. • Detecting: • abnormal behaviour • new kind of customers • etc. Anomaly detection Copyright Dragons Out Oy and Gofore 2025 Figure: Example of time series 13
- 14. • NLP can be used to analyze text within different defect reports to identify areas of affected functionality • Clustering algorithms such as SVM are used to define defect categories • Text similarity metrics are used to identify similar or duplicate defects • Particularly useful for automated defect reporting systems • Case: MS Windows and Firefox and on large projects with many software engineers Large project defect classification Copyright Dragons Out Oy and Gofore 2025 14 Source: STA Consulting
- 15. • ML models can be trained to identify those defects most likely to cause critical system failures from automatically generated defect reports Defect prioritization Copyright Dragons Out Oy and Gofore 2025 15 Ref: Kim, D.; Wang, X.; Kim, S.; Zeller, A.; Cheung, S.C.; Park, S. (2011). “Which Crashes Should I Fix First? Predicting Top Crashes at an Early Stage to Prioritize Debugging Efforts,” in the IEEE Transactions on Software Engineering, volume 37 Source: STA Consulting
- 16. Prediction Copyright Dragons Out Oy and Gofore 2025 16
- 17. • Where should I start testing? • Which tests will give me results fastest? • Do I need all my 10 000 test cases / scripts? • Which parts of software are likely to have defects? Tester’s objectives Copyright Dragons Out Oy and Gofore 2025 17
- 18. • Go through functionalities • Identify risky areas • Focus testing • Case: Eggplant • https://www.eggplantsoftw are.com/ Predict quality issues Copyright Dragons Out Oy and Gofore 2025 18
- 19. • Analyze risk in commits to the code • Select the tests that test the risky areas • Reduce the number of test cases needed to run • Case: Appsurify • https://appsurify.com/ Select most suitable test cases Copyright Dragons Out Oy and Gofore 2025 19
- 20. • Power of open source developers • Machine learning algorithms learn from the community automatically • Analyse code and propose improvements • Case: DeepCode by Snyk • https://www.deepcode.ai/ Code analysis using AI Copyright Dragons Out Oy and Gofore 2025 20
- 21. Fault-tolerant test automation Copyright Dragons Out Oy and Gofore 2025 21
- 22. • How can I get my regression test cases pass more often when the software under test changes? • Can I use test automation scripts from another similar project? • Can I use some generic test without my own scripting? Tester’s objectives Copyright Dragons Out Oy and Gofore 2025 22
- 23. • AI-based test generation • Checking functional differences • Checking visual differences • Automatically learning the test automation • For: regression testing after changes • Case: retest • https://retest.de/ai-based-test- generation/ AI with regression testing Copyright Dragons Out Oy and Gofore 2025 23
- 24. AI-enabled test creation from models • Automatically scope and generate optimized test cases • Generate test scripts for automated execution from models and manual tests • Display the test coverage requirements • Using Symbolic AI • Integrate with existing tools • Case: Conformiq • https://www.conformiq.com/ CONFIDENTIAL 24
- 25. • Python-editor PyCharm can use Dev Tools AI library • Asks which element to click on the screen and then keeps it running • Suggests how to fix semiautomatically a changed test script that doesn’t pass any more • Reduces the need for test script maintenance • Supports e.g., Cypress, Selenium, Appium, Playwright • https://www.dev-tools.ai/ AI library helps pass Python test scripts Copyright Dragons Out Oy and Gofore 2025 25
- 26. • Build tests through interface backed by ML • Plain English • New life for “record-playback” • ML algorithms maintain the tests • Case: Functionize • https://www.functionize.com/ Intelligent testing Copyright Dragons Out Oy and Gofore 2025 26
- 27. • Supports test automation object recognition • Helps with typical GUI recognition problems for test automation • Can be used as a support library for existing test automation • Cases: • ImageHorizonLibrary is a cross-platform library for Robot Framework. • Eficode, Oulu University and Business Finland • https://www.eficode.com/projects/testomat Image recognition with AI for open source test automation Copyright Dragons Out Oy and Gofore 2025 27
- 28. Copyright Dragons Out Oy and Gofore 2025 AI Assisted Test Automation • Setup • Robot Framework with custom Libraries for AI, Jira and Data collection • OpenAI GPT4 GenAI with RAG and Assistants • Experiments • Automatic test creation with AI based on the source code • “Self-healing” test automation with the help of AI • Test failure automatic pre-analysis with AI • Results • Increased test coverage and harmonized test cases • Self-healing is dangerous so more of “solution proposals” than self-healing • Extremely fast failure analysis and case explanations • Case Gofore • Some of the early AI experiments and POCs that proved to bring value for the automation Run test Gather info Analyze error Create solution Create Test Done CI/CD Ticket Review Yes No Delete Yes No Accept = Human Decision = AI Action = Trad. Automation Pass 28
- 29. Faster test automation scripting with AI assistants Copyright Dragons Out Oy and Gofore 2025 29
- 30. • Can LLMs / SLMs / RAG / Generative AI help me? • I want to produce test scripts faster • I want ideas for my test cases / scripts Tester’s objectives Copyright Dragons Out Oy and Gofore 2025 30
- 31. • Python-editor PyCharm has Refact.ai library that uses ChatGPT • Understands Python and Robot Framework test automation setup • Refact.ai is labelled as an AI coding assistant software • https://refact.ai/ Python enhanced with ChatGPT Copyright Dragons Out Oy and Gofore 2025 31
- 32. • Assist test automation script creation • Suggest what the user might want • What kind of code • What kind of test script • User reviews and approves suggestions • Based on Generative AI • https://github.com/features/copilot Github Copilot Copyright Dragons Out Oy and Gofore 2025 32
- 33. Playwright MCP • Playwright MCP Using Github Copilot to generate test scripts • Access to multiple LLMs, e.g., Claude • Based on test scenarios • Within the editor (e.g., Visual Studio Code) • https://executeautomation.github.io/mcp-playwright/docs/intro Copyright Dragons Out Oy and Gofore 2025 33
- 34. Generative AI (SLM/LLM) helps in test creation • Derive tests from specifications • Create tests and test data based on user input • Enabled by good prompt engineering • Challenge: repeated teaching • User reviews AI proposals and uses in their tests • https://chat.openai.com/ (ChatGPT) • https://gemini.google.com/ • https://copilot.microsoft.com/ (Bing) • https://claude.ai/ 3.5 Sonnet • https://chat.mistral.ai/chat Le Chat • https://www.deepseek.com/ Deepseek
- 35. Efficiency warning! • Without professional-level coding/testing skills and a review attitude the AI effect can be the opposite of what is targeted • “Code analysis firm Gehtsoft USA sees no major benefits from AI dev tool when measuring key programming metrics, though others report incremental gains from coding copilots with emphasis on code review.” • Devs gaining little (if anything) from AI coding assistants at CIO magazine Sep 26, 2024 Copyright Dragons Out Oy and Gofore 2025 35
- 36. AI policy or guidance • Generative AI requires data as input to give relevant answers • Many organizations have strict confidentiality, privacy, or other reasons to limit what information can be given to AI assistants or Generative AI • They may need to limit using external large LLMs, even internally or at least with their customers • Each company should have an AI policy or guidance created and enforced to guide employees on AI usage Copyright Dragons Out Oy and Gofore 2025 36
- 38. Autonomous testing Copyright Dragons Out Oy and Gofore 2025 38
- 39. • Can I have some tests done before I start my test scripting? • Can I leave testing to AI? • Are there some default tests that fit my product testing needs? Tester’s objectives Copyright Dragons Out Oy and Gofore 2025 39
- 40. • Default tests to web sites • Teach your own tests • Running on containers in virtual machines • Use for regression testing, cross- browser testing • Case: Mabl • https://www.mabl.com/ Auto-healing automated testing of web Copyright Knowit Solutions Oy 2021 40
- 41. • Extending existing test automation sets • With e.g. Robot Framework, Playwright, Selenium, Cypress • Use tests as self-driving test-bots • Exploring web apps • Regression testing • Autonomous testing • Case: https://wopee.io/ Test automation assistants and bots Copyright Dragons Out Oy and Gofore 2025 41
- 42. AI-augmented test automation solution • Digital twin simulation • Software development in the loop • Hardware development in the loop • Testing in the loop • AI in the loop • AI techniques streamline and enhance the testing by automatically identifying and executing test cases. • Customized language models (SLM & LLM) and digital twin technology to create a comprehensive and adaptive testing environment. • Case • Gofore’s IntelliTestAI- Intelligent Test Automation solution for Enhanced Quality • www.gofore.com Copyright Dragons Out Oy and Gofore 2025 42
- 43. AI integrated into all testing (AI augmented testing) Copyright Dragons Out Oy and Gofore 2025 43
- 45. Resources Copyright Dragons Out Oy and Gofore 2025 45
- 46. • Trainings and certifications • https://www.istqb.org/certification-path-root/ai-testing.html • Standards • ISO • AI/ML standard https://www.iso.org/standard/74438.html • Testing of AI-based systems https://www.iso.org/standard/79016.html • IEEE • What is AI Software Testing https://ieeexplore.ieee.org/document/8705808 • AI testing perspectives https://ieeexplore.ieee.org/document/9514942 • Application of AI in Testing https://ieeexplore.ieee.org/document/9676244 Promoting and teaching AI and testing Copyright Dragons Out Oy and Gofore 2025 46
- 47. • Devs gaining little (if anything) from AI coding assistants • https://www.cio.com/article/3540579/devs-gaining-little-if-anything-from-ai-coding- assistants.html • AI in Testing: Impact, Problems, Challenges and Prospect • https://www.researchgate.net/publication/357876318_Artificial_Intelligence_in_Software _Testing_Impact_Problems_Challenges_and_Prospect • Utilizing AI in Software Testing • https://www.theseus.fi/handle/10024/263992 • AI Applied to Software Testing • https://dl.acm.org/doi/10.1145/3616372 • AI Applied to Testing: A Literature Review • https://ieeexplore.ieee.org/abstract/document/9141124 • ChatGPT helps testing • https://www.linkedin.com/pulse/gpt-4-sdlcs-secret-weapon-reinventing-testing-phase-andy- abbott/ Some more research notes Copyright Dragons Out Oy and Gofore 2025 47
- 48. Conclusion 3 Copyright Dragons Out Oy and Gofore 2025 48
- 49. • AI-enabled testing is already a reality • Many companies are enhancing their solutions with AI • New companies are set up around AI • AI can provide simplicity to complex software development projects. • With Generative AI, manual and automated test script creation is more productive • With AI, testing activities can be focused on high-risk areas • With AI, test automation becomes more autonomous • Testers are freed to create new tests • Start your AI journey today, if you haven’t already Conclusion Copyright Dragons Out Oy and Gofore 2025 49
- 50. Any questions? Follow and share the Kari’s testing book projects: • https://www.dragonsout.com • https://www.act2lead.net/ Social media • Kari https://www.linkedin.com/in/karikakkonen/ • Dragons Out https://www.facebook.com/DragonsOutOy Ask questions: kari.kakkonen@dragonsout.com kari.kakkonen@gofore.com 50 Copyright Dragons Out Oy and Gofore 2025