How AI supports software testing Kari Kakkonen at Upload

How AI supports
software testing and
test automation
Copyright
Dragons
Out
Oy
2024
• Kari Kakkonen
• https://www.linkedin.com/in/karikakkonen
• Gofore Verify Oy
• Upload
• Helsinki
• 10.12.2024
1

ROLES
• Gofore Verify Oy, Expertise Capability Owner
• Dragons Out Oy, CEO, Children’s and testing
author
• TMMi, Board of Directors
• Treasurer of Finnish Software Testing Board
(FiSTB)
ACHIEVEMENTS
• Tester of the Year in Finland 2021
• EuroSTAR Testing Excellence Award 2021
• Exemplary DevOps Instructor Award 2023 by
DASA
• ISTQB Executive Committee 2015-2021
• Influencing testing since 1996
• Ranked in 100 most influential IT persons in
Finland (Tivi magazine)
• Great number of presentations in Finnish and
international conferences
• TestausOSY/FAST founding member.
• Author of Dragons Out testing book
• Co-author of ACT2LEAD software testing
leadership handbook
• Co-author of Agile Testing Foundations book
• Regular blogger in Tivi-magazine
Kari Kakkonen, Expertise Capability Owner
SERVICES
• ISTQB Advanced, Foundation, Agile Testing, AI Testing
• Quality Professional
• DASA DevOps, Agile, Scrum, Product Management
• Quality & Test process and organization development,
Metrics, TMMi and other assessments
• Agile testing, Scrum, Kanban, Lean
• Leadership
• Test automation, Mobile, Cloud, DevOps, AI
• Quality, cost, benefits
EDUCATION
• ISTQB Expert Level Test Management & Advanced Full
& Agile Tester certified
• DASA DevOps, Scrum Master and SAFe certified
• TMMi Professional, Assessor, Process Improver certified
• SPICE provisionary assessor certified
• M.Sc.(Eng), Helsinki University of Technology (present Aalto
University), Otaniemi, Espoo
• Marketing studies, University of Wisconsin-Madison,
the USA.
BUSINESS DOMAINS
Wide spread of business domain knowledge: Embedded,
industry, public, training, telecommunications, commerce,
Insurance, banking, pension.
2
www.gofore.com
www.dragonsout.com
www.act2lead.net
MORE INFORMATION
linkedin.com/in/karikakkonen/
Copyright Dragons Out Oy 2024

The book project ”Dragons Out!”
• Mission
• “Software testing brought to children”
• Book
• Author Kari Kakkonen
• Illustrator Adrienn Széll
• Text and illustration rights Dragons Out Oy
• In Finnish, English, Polish, French and growing
• For ages of 10-99
• Free “Dragon lesson in software testing”
presentation under Creative Commons –
license
• Translated to 20 languages!
• More info: www.dragonsout.com
4

ACT 2 LEAD software testing leadership handbook
● Easy to read - chapters can be read in any order.
● Structure: questions, answers and cases.
○ 34 main chapters = questions, see next page.
● 270+ pages,
○ in Finnish (softback, e-book)
○ in English (e-book).
● For people like CxO, director, head of, manager, product
owner, designer, developer, test manager, tester and
student.
● Teaches to lead testing, not to test.
Buy the book: https://leanpub.com/act2lead
Book website: www.act2lead.net 5

ISTQB GLOBAL PRESENCE
• Number of exams
administered: over 1,3 million
• Number of certifications
issued: 957,000
• In 130 countries
Copyright Dragons Out Oy 2024 6

TMMi for test improvement in all kinds of testing,
including agile and DevOps

• Why we need AI
• How AI works in the NASA case
• Journey through AI opportunities for
testing
Agenda

Why AI right now?
Four drivers behind AI revolution
9
Computation growth due to general purpose GPUs The rise of Big data
Community based achievements in Deep learning Open source tools and frameworks

Testing needs AI
• Software complexity increases
• Quantity of software increases
• Software development (including testing)
needs to be faster
• Faster time to market
• DevOps / Continuous Delivery
• Test automation is maintenance heavy
• Even exploratory testing takes too much time
• AI brings complexity that can only be solved
by AI

AI augmented testing
• Gartner term
• https://www.gartner.com/en/documents/5194063
• “Software engineering leaders are now prioritizing development
productivity to enhance market responsiveness and build software
applications more efficiently, while also aiming to maintain high
quality. To meet this challenge they are increasingly turning to AI-
augmented testing tools.”

My journey into AI testing 2016-2024
• ISTQB conference in Korea – Stuart Reid keynote – wake up
• PhD in AI Juhani Teeriniemi to build AI trainings
• AI testing tool research and experiments
• A4Q AI and Software Testing – Adam Leon Smith
• Open-source test automation findings to match Robot Framework
• ISTQB AI testing – Klaudia Dussa-Zieger
• AI testing bots
• GenAI talks and piloting
• Knowit Quality Summit with AI testing theme
• Gofore - Independent AI testing in a simulation
• Gofore – AI integrated into all testing

How AI works in a NASA case

NASA’s code review
• In static testing, several machine
learning algorithms can be used to
measure code quality.
Figure: Margaret Hamilton standing next
to source code of guidance system of
Apollo spacecraft.
www.nasa.gov/

• Several code quality metrics are currently
being used. The most popular ones have
been introduced in 1970.[1,2]
1 T.J. McCabe, A Complexity Measure, IEEE Transactions on Software Engineering (1976) SE-2, 4, 308.
2 M.H. Halstead, Elements of Software Science, Elsevier (1977).
3 N.E. Fenton, S.L. Pfleeger, Software Metrics: A Rigorous & Practical Approach, International Thompson Press (1997).
4 S.J. Sayyad, T.J. Menzies, The PROMISE Repository of Software Engineering Databases (2005).
• Goal is to provide feedback:
1. for developers about code quality.
2. for testers about needed test coverage.
3. for management about the development
project
• Although traditional metrics are extensively
used, benefit of using them has been
questioned.[3]
0
50
100
150
200
250
300
350
400
450
0 20 40 60 80 100 120
Number
of
lines.
Cyclomatic complexity
Scattering of defects in NASA CM1 project
OK Defected
Figure: NASA Metrics Data Program, CM1 Project[4]
In real applications defected modules do not separate from
non-defected by standard metrics.
AI code quality metrics

1
2
𝑤 2
+ 𝐶 ෍
𝑖=1
𝑁
ξ𝑖
1-𝑦𝑖 𝑤, 𝑓(𝑥𝑖) + 𝑏 ≤ ξ𝑖
Minimize
Subject to with all values of i.
Quality metric 1
Quality
metric
2
• Machine learning approach was tested for NASA’s CM1
software repository[1].
• Support vector type of machine learning algorithm was
developed for this demonstration.
• This task can be solved by supervised machine learning
methods because both input (quality metrics) and output
(defects) is known.
• Goal is to find such hyperplane in multidimensional data space
that separate defected modules from non-defected.
• Mathematically, problem can be described with algorithm:
AI-guided software testing with NASA data

0,0
0,2
0,4
0,6
0,8
Tested on real data against NASA CM1 project[1]
Performance
(F)
Training set: 1595
Test set: 400
𝐹 =
2 ∗ 𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 ∗ 𝑟𝑒𝑐𝑎𝑙𝑙
𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 + 𝑟𝑒𝑐𝑎𝑙𝑙
AI-guided software testing with NASA data

Anomaly detection

• Can I find defects faster from lots of test runs?
• Which automatic defect reports / crash reports are actual defects?
• Are there duplicate reports?
• Automation to defect reporting?
Tester’s objectives

• Detecting:
• abnormal behaviour
• new kind of customers
• etc.
Anomaly detection
Figure: Example of time series
21

• NLP can be used to analyze text within different defect reports to
identify areas of affected functionality
• Clustering algorithms such as SVM are used to define defect
categories
• Text similarity metrics are used to identify similar or duplicate defects
• Particularly useful for automated defect reporting systems
• Case: MS Windows and Firefox and on large projects with many
software engineers
Large project defect classification
Source: STA Consulting

• ML models can be trained to identify those defects most likely to
cause critical system failures from automatically generated defect
reports
Defect prioritization
Ref: Kim, D.; Wang, X.; Kim, S.; Zeller, A.; Cheung, S.C.; Park, S. (2011). “Which Crashes Should I Fix First? Predicting Top Crashes at an
Early Stage to Prioritize Debugging Efforts,” in the IEEE Transactions on Software Engineering, volume 37
Source: STA Consulting

Prediction

• Where should I start testing?
• Which tests will give me results
fastest?
• Do I need all my 10 000 test cases /
scripts?
• Which parts of software are likely
to have defects?

AI
AI review
Requirement
management
Code quality
analysis
Version
control
Test
management
Defect
management

Developer
Product
Owner
Scrum team
Management
AI
AI review

• Goal is to understand technical quality and test coverage of
modules. Also, goal is to guide and focus testing:
• Focusing technical debt reduction
• Found defects per module
• Guide for test coverage in automating system level testing.
Requires that there is link from components (version control) and tests (test management) to feature descriptions and from reported defects to modules (observation control)
Module Developer Benchmark
(order.numb.)
Relative risk Number of
reported bugs
Module_1 Jaakko 50 58 4
Module_2 Jaakko 4 5 1
Module_3 Mikko 180 95 7
SCRUM TEAM
28

• Goal is to provide overview:
• How fast the development is advancing
• Maturity for release
• Quality and technical debt of the software
• Defect probabilities
• Risks related to modifications of the
software.
Interpretation:
• Module 5 has high defect probability and is
related to five most important features of
the application. This results into high risk in
release at this moment.
• Defect probability of module 10 has fallen
to acceptable level.
Start-up
phase
Development
phase
Maturing
phase
MANAGEMENT
29

• Go through functionalities
• Identify risky areas
• Focus testing
• Case: Eggplant
• https://www.eggplantsoftw
are.com/
Predict quality issues

• Analyze risk in commits to the code
• Select the tests that test the risky areas
• Reduce the number of test cases needed to run
• Case: Appsurify
• https://appsurify.com/
Select most suitable test cases

• Power of open source
developers
• Machine learning
algorithms learn from the
community automatically
• Analyse code and propose
improvements
• Case: DeepCode
• https://www.deepcode.ai/
Code analysis using AI

Fault-tolerant test automation

• How can I get my regression test
cases pass more often when the
software under test changes?
• Can I use test automation scripts
from another similar project?
• Can I use some generic test without
my own scripting?

• AI-based test generation
• Checking functional differences
• Checking visual differences
• Automatically learning the test
automation
• For: regression testing after changes
• Case: retest
• https://retest.de/ai-based-test-
generation/
AI with regression testing

• Python-editor PyCharm can use Dev
Tools AI library
• Asks which element to click on the
screen and then keeps it running
• Suggests how to fix
semiautomatically a changed test
script that doesn’t pass any more
• Reduces the need for test script
maintenance
• Supports e.g., Cypress, Selenium,
Appium, Playwright
• https://www.dev-tools.ai/
AI library helps pass Python test scripts

• Build tests through interface backed by ML
• Plain English
• New life for “record-playback”
• ML algorithms maintain the tests
• Case: Functionize
• https://www.functionize.com/
Intelligent testing

• Supports test automation object recognition
• Helps with typical GUI recognition problems for test automation
• Can be used as a support library for existing test automation
• Cases:
• ImageHorizonLibrary is a cross-platform library for Robot Framework.
• Eficode, Oulu University and Business Finland
• https://www.eficode.com/projects/testomat
Image recognition with AI
for open source test automation

AI Assisted Test Automation
• Setup
• Robot Framework with custom Libraries for AI,
Jira and Data collection
• OpenAI GPT4 GenAI with RAG and Assistants
• Experiments
• Automatic test creation with AI based on the
source code
• “Self-healing” test automation with the help of AI
• Test failure automatic pre-analysis with AI
• Results
• Increased test coverage and harmonized test
cased
• Self-healing is dangerous so more of “solution
proposals” than self-healing
• Extremely fast failure analysis and case
explanations
• Case Gofore
• Some of the early AI experiments and POCs that proved to bring
value for the automation
Run test
Gather info
Analyze
error
Create
solution
Create Test
Done
CI/CD
Ticket
Review
Yes
No
Delete
Yes
No
Accept
= Human Decision
= AI Action
= Trad. Automation
Pass

Faster test automation scripting
with AI assistants

• Can LLMs / SLMs / RAG /
Generative AI help me?
• I want to produce test scripts
faster
• I want ideas for my test cases /
scripts

• Python-editor PyCharm has
Refact.ai library that uses
ChatGPT
• Understands Python and Robot
Framework test automation setup
• Refact.ai is labelled as an AI
coding assistant software
• https://refact.ai/
Python enhanced with ChatGPT

• Assist test automation script
creation
• Suggest what the user might want
• What kind of code
• What kind of test script
• User reviews and approves
suggestions
• Based on Generative AI
• https://github.com/features/copilot
Github Copilot

• Derive tests from specifications
• Create tests and test data based on
user input
• Enabled by good prompt engineering
• Challenge: repeated teaching
• User reviews AI proposals and uses in
their tests
• https://chat.openai.com/ (ChatGPT)
• https://gemini.google.com/
• https://copilot.microsoft.com/ (Bing)
• https://claude.ai/ 3.5 Sonnet
Generative AI (SLM/LLM) helps in test creation

Efficiency warning!
• Without professional-level coding/testing skills and a review attitude
the AI effect can be the opposite of what is targeted
• “Code analysis firm Gehtsoft USA sees no major benefits from AI
dev tool when measuring key programming metrics, though others
report incremental gains from coding copilots with emphasis on code
review.”
• Devs gaining little (if anything) from AI coding assistants at CIO
magazine Sep 26, 2024

AI policy or guidance
• Generative AI requires data as input to give relevant answers
• Many organizations have strict confidentiality, privacy, or other
reasons to limit what information can be given to AI assistants or
Generative AI
• There may need to limit using external large LLMs, even internally or
at least with their customers
• Each company should have an AI policy or guidance created and
enforced to guide employees on AI usage

Autonomous testing

• Can I have some tests done
before I start my test scripting?
• Can I leave testing to AI?
• Are there some default tests that
fit my product testing needs?

• “World’s first”
• Go automatically through mobile apps
• Use general test cases
• Reinforcement learning
• Case: test.ai
• https://www.test.ai/
AI-powered mobile test automation platform

• Assist extending existing
test automation sets
• With e.g. Robot Framework,
Playwright, Selenium,
Cypress
• Use tests as self-driving
test-bots
• Exploring web apps
• Regression testing
• Autonomous testing
• Case: https://wopee.io/
Test automation assistants and bots

AI-augmented test automation solution
• Digital twin simulation
• Software development in the loop
• Hardware development in the loop
• Testing in the loop
• AI in the loop
• AI techniques streamline and enhance the testing by automatically identifying
and executing test cases.
• Customized language models (SLM & LLM) and digital twin technology to create a
comprehensive and adaptive testing environment.
• Case
• Gofore’s IntelliTestAI- Intelligent Test Automation solution for Enhanced
Quality
• www.gofore.com Copyright Dragons Out Oy 2024 51

AI integrated into all testing
(AI augmented testing)

•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Read more
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•

Resources

• Trainings and certifications
• https://www.istqb.org/certification-path-root/ai-testing.html
• Standards
• ISO
• AI/ML standard https://www.iso.org/standard/74438.html
• Testing of AI-based systems https://www.iso.org/standard/79016.html
• IEEE
• What is AI Software Testing https://ieeexplore.ieee.org/document/8705808
• AI testing perspectives https://ieeexplore.ieee.org/document/9514942
• Application of AI in Testing https://ieeexplore.ieee.org/document/9676244
Promoting and teaching AI and testing

• Devs gaining little (if anything) from AI coding assistants
• https://www.cio.com/article/3540579/devs-gaining-little-if-anything-from-ai-coding-
assistants.html
• AI in Testing: Impact, Problems, Challenges and Prospect
• https://www.researchgate.net/publication/357876318_Artificial_Intelligence_in_Software
_Testing_Impact_Problems_Challenges_and_Prospect
• Utilizing AI in Software Testing
• https://www.theseus.fi/handle/10024/263992
• AI Applied to Software Testing
• https://dl.acm.org/doi/10.1145/3616372
• AI Applied to Testing: A Literature Review
• https://ieeexplore.ieee.org/abstract/document/9141124
• ChatGPT helps testing
• https://www.linkedin.com/pulse/gpt-4-sdlcs-secret-weapon-reinventing-testing-phase-andy-
abbott/
Some more research notes

Conclusion v1

Conclusion 2

Conclusion 3

• AI-enabled testing is already a reality
• Many companies are enhancing their solutions with AI
• New companies are set up around AI
• AI can provide simplicity to complex software development projects.
• With Generative AI, manual and automated test script creation is
more productive
• With AI, testing activities can be focused on high-risk areas
• With AI, test automation becomes more autonomous
• Testers are freed to create new tests
• Start your AI journey today, if you haven’t already
Conclusion

Any questions?
Follow and share the Kari’s testing book projects:
• https://www.dragonsout.com
• https://www.act2lead.net/
Social media
• Kari https://www.linkedin.com/in/karikakkonen/
• Dragons Out https://www.facebook.com/DragonsOutOy
Ask questions:
kari.kakkonen@dragonsout.com
kari.kakkonen@gofore.com
61

How AI supports software testing Kari Kakkonen at Upload

More Related Content

Similar to How AI supports software testing Kari Kakkonen at Upload (20)

More from Kari Kakkonen (20)

Recently uploaded (20)

How AI supports software testing Kari Kakkonen at Upload