How to apply AI to Testing

@roesslerj1
How to apply AI to Testing
Dr. Jeremias Rößler
https://retest.de

@roesslerj2
• “99% of respondents face some kind of challenge with
testing in agile development.”
• “Test automation tools are fundamental to achieving
the continuous testing approach required by DevOps.”
• “by 2021, 50% of enterprises will leverage intelligent
test automation driven by AI and machine learning.”
*Source: Gartner 2017, Magic Quadrant for Software Test Automation

@roesslerj3
Problem: Software Testing ist komplex und erzeugt viel manuellen Aufwand
Trends: DevOps, Agile, Shift Left
32%
34%
34%
35%
36%
37%
35%
40%
45%
The Automation Challenges
Capgemini: 40% of the total IT budget will be allocated to QA
+ Testing in 2019.
Ideate/
analyze/plan Design/
Develop/Build
Test
Release
High Interest
Medium Interest
Low Interest
Source: Forrester Research, Inc
We don't have the right automation tools
Challenge with the Test Data and Environment Availability
Delivery methodology doesn't support test automation
We have difficulties to integrate the different automation
tools together
Application functionality changes too frequently
We don't have the right automation testing
process/method
Lack of stability of test script
Lack of skilled and experience test automation resource
We have difficulties to automate because of use of multiple
development lifecycles
*Source: World Quality Report 2016-17

@roesslerj6
1 - 2 = ?
• -1 is mathematically correct
• 0 if I don’t want to payback gift certiﬁcates to costumers
• 23 if I’m working with hours
• 59 if I’m working with minutes or seconds
• Something between 27 and 30 if I’m working with days
• 11 if I am working with months
• exception if I am working with degrees Kelvin

@roesslerj7
complexity
UITester Software
easy-to-use
intuitive
simple

@roesslerj8
component recognition
password (text)
username (text)
enter text (action)
click button (action)
complexity
UITester Software

@roesslerj9
Solution-Knowledge
Code
test
formalize
Tester
complexity

@roesslerj13
How do you bring AIinto testing?

@roesslerj14
How do YOU do Testing?

@roesslerj15
Testing
Actual wrong
Bug
Comparison Expected vs Actual
Test Execution
Done
No Difference

@roesslerj16
Actual wrong
Bug
Update Expected
Expected wrong
Test Execution
Done
No Difference Difference
Evaluate Difference
Testing

@roesslerj17
When is a bug a bug?
When it’s not a feature!

@roesslerj18
Is it a bug?
.
..
text.c
other.files
for (. . .) {
if (name[0] == '.') continue;
. . .
}
ls

@roesslerj21
What is a bug?
Without speciﬁcation, there are no bugs
— only surprises.
Brian Kernighan
“

@roesslerj22
Why do we test?
After
implementation?
1 + 1 = 3
2 + 2 = 4

@roesslerj26
Why do we test?
After
implementation?
1 + 1 = 3
2 + 2 = 4
After
change?
1 + 1 = 3
1 + 1 = 5

@roesslerj
automated
regression test
≠
test
27

@roesslerj
automated
regression test
=
version control
28

@roesslerj29
Why do we test?
After
implementation?
1 + 1 = 3
2 + 2 = 4
After
change?
1 + 1 = 3
1 + 1 = 5
1 + 1 = 2

@roesslerj31
Actual wrong
Bug
Update Expected
Expected wrong
Test Execution
Done
Evaluate Difference
Testing

@roesslerj32
Actual wrong
Bug
Update Expected
Expected wrong
Test Execution
Done
Evaluate Difference
Testing
effort

@roesslerj33
@Test
public void testReTestWebSiteClassic() throws Exception {
driver.get("https://retest.de");
assertThat(driver.getTitle()).isEqualTo("retest");
WebElement toDownload = driver.findElement(By.name("submit"));
toDownload.click();
assertTrue(driver.findElement(By.id("errorMsg")).isDisplayed());
}

@roesslerj35
public void testSomething() throws Exception {
// here be some test code ...
UIElement element = driver.getElement(“path/to/element”);
assertEquals(null, element.getAction());
assertEquals(“action”, element.getActionCommand());
assertEquals(true, element.isEnabled());
assertEquals(true, element.isFocusable());
assertEquals(“Lucida Grande”, element.getFont().getName());
assertEquals(13, element.getFont().getSize());
assertEquals(0, element.getFont().getStyle());
assertEquals(23, element.getHeight());
assertEquals(null, element.getIcon());
assertEquals(0, element.getMnemonic());
assertEquals(null, element.getPressedIcon());
assertEquals(true, element.isSelected());
assertEquals(“label”, element.getText());
assertEquals(119, element.getWidth());
assertEquals(27, element.getX());
assertEquals(191, element.getY());
}

@roesslerj42
Actual wrong
Bug
Update Expected
Expected wrong
Test Execution
Done
Evaluate Difference
Testing
effort
effort

@roesslerj48
https://github.com/retest/recheck-web

@roesslerj49
@Test
public void testReTestWebSite() throws Exception {
driver.get("https://retest.de");
re.startTest();
re.check(driver, "index");
WebElement toDownload = driver.findElement(By.name("submit"));
toDownload.click();
re.check(driver, "download_1");
re.capTest();
}
v

@roesslerj52
// Use the RecheckDriver as a wrapper for your usual driver
driver = new RecheckDriver( new ChromeDriver() );
// Use the unbreakable recheck implementation.
re = new RecheckWebImpl();
// Single call instead of multiple assertions
re.check( driver, "index" );
// Will issue a warning, as the id has changed
driver.findElement( By.id( "intro-slider" ) );

@roesslerj53
*************** recheck warning ***************
The HTML id attribute used for element identification changed from 'intro-
slider' to 'introSlider'.
retest identified the element based on the persisted old state.
If you apply these changes to the state , your test will break.
Use `By.id("introSlider")` or `By.retestId("9c40281d-5655-4ffa-9c6d-
d079e01bb5a3")` to update your test.

@roesslerj54
Beneﬁts of recheck:
- Reduce effort to create tests
- Reduce effort to maintain tests
- Tests are more complete
- Tests are less fragile

@roesslerj55
How do you bring AIinto testing?

@roesslerj59
Inﬁnite-Monkey-Theorem:
A monkey hitting keys at random on a typewriter
for a long enough time
will type the complete works of William Shakespeare.

@roesslerj60
We replace the typewriter with a computer…

@roesslerj61
public static void main(String... args) throws Exception {
Robot robot = new Robot();
while (true) {
robot.mouseMove(random.nextInt(maxX), random.nextInt(maxY));
robot.mousePress(InputEvent.BUTTON1_DOWN_MASK);
robot.mouseRelease(InputEvent.BUTTON1_DOWN_MASK);
robot.delay(200);
for (char inputChar : randomString().toCharArray()) {
robot.keyPress((int) inputChar);
robot.keyRelease((int) inputChar);
robot.delay(10);
}
robot.keyPress(KeyEvent.VK_ENTER);
robot.keyRelease(KeyEvent.VK_ENTER);
}
}
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.

@roesslerj64
- gremlins.js
- NetFlix Chaos Monkey
- UI/Application Exerciser Monkey
Open Source Products

@roesslerj
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14. 67
public static void main(String... args) throws Exception {
WebDriver driver = new FirefoxDriver();
driver.get("http://www.retest.de");
while (true) {
List<WebElement> links = driver.findElements(By.tagName("a"));
links.get(random.nextInt(links.size())).click();
Thread.sleep(500);
List<WebElement> fields =
driver.findElements(By.xpath("//input[@type='text']"));
WebElement field = fields.get(random.nextInt(fields.size()));
field.sendKeys(randomString());
Thread.sleep(500);
}
}

@roesslerj68
Intelligent Monkey
DEMO

@roesslerj73
Problem Space
Problem
Space

@roesslerj79
train
AI
Artiﬁcial
Neural Net
Genetic
Algorithm
AI

@roesslerj80
https://code.fb.com/developer-tools/sapienz-intelligent-automated-software-testing-at-scale/

@roesslerj81
test
1.0
approve
1.1
diff
train
user
automate

@roesslerj83
https://code.fb.com/developer-tools/finding-and-fixing-software-bugs-automatically-with-sapfix-and-sapienz/
Automatically Fixing Bugs

@roesslerj86
AI
read-back
easy-to-use
intuitive
simple

@roesslerj
@roesslerjX
test
1.0
approve
1.1
diff
train
user
automate
@roesslerjX
https://github.com/retest/recheck-web
87
Summary
Difference Testing
little maintenance
no wasteful testing
yet more complete
Monkey-Testing:
cheaper
faster
better
multiplies manual tests

@roesslerj
roessler@retest.de
@roesslerj
Questions?
https://www.facebook.com/retest.de
http://retest.rocks

How to apply AI to Testing

More Related Content

What's hot (20)

Similar to How to apply AI to Testing (20)

More from SAP SE (12)

Recently uploaded (20)

How to apply AI to Testing