SlideShare a Scribd company logo

20200630 Rakuten QA meetup #2 "Improve test automation operation"

Download as PPTX, PDF
S
Sadaaki Emura

Test automation brings benefits but also struggles like many failure reports. Currently, temporary unstable failures make up 73.4% of issues but investigating them wastes time. The presenter proposes an auto recover system to automatically re-run tests predicted to fail due to temporary issues. This could reduce wasted monthly operation time from 75 hours to around 25 hours by automating recovery of the most common failure type. The system would use OCR, screenshots and previous data to predict failure reasons and automatically re-run tests to improve test automation efficiency.

Engineering
1 of 26
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Improve test automation operation June 30th, 2020 Sadaaki, Emura Leisure Product Dept. Rakuten, Inc.
2 Who am I Name : Sadaaki Emura (nickname M) Join in Rakuten : 2007 Career : Embedded engineer (2000-2002) Web engineer (2002-2015) Product manager (2015-2016) QA (2016~) Role: Test Automation Engineer Lead Birthplace : Kanazawa-city Hobby: jog , climbing , horse racing
3 Organization Developer group QA group Manual test team Test automation team @IllustAC ココ の人 :5 services
4 Organization
5 Today story 1. Test Automation operation is struggling 2. Improve test automation operation
6 Test automation brings benefits in proportion to coverage! • Reduce total cost of regression test • Quick feedback • Take work easy © FREEIMAGES
7 Test automation brings struggle actually ©photoAC focus • Many failure reports come • Synchronize latest specification • a few test automation engineers
8 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable Failure report classif y
9 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable • Good activity ! • Happy to report engineer! ©いらすとや
10 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable • Script has bug • Script covers old specification • Scenario dependency ©いらすとや
11 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable • Test data invalid • Application is not ready ©いらすとや
12 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable • Network disconnected • 500 error (releasing operation?) • 503 error (what’s happened?) ©いらすとや
13 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable
14 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable 0… 5.3 21.1 73.4 Fing bug Test automation script issue Environment issue Temporary unstable
15 0… 5.3 21.1 73.4 Operation for Temporary unstable 1.Investigate failure report (common) 2.Run test automation script again 3.Check if result is success Ex. Operation Cost 1 operation = 2 mins Daily operation = 100 * 2 = 200 mins Monthly operation = 200 * 22 = 75 hours Temporary Unstable Waste time!
16 Today story 1. Test Automation operation is struggling 2. Improve test automation operation
17 0… 5.3 21.1 73.4 Operation for Temporary unstable 1.Investigate failure report (common) 2.Run test automation script again 3.Check if result is success Ex. Operation Cost 1 operation = 2 mins Daily operation = 100 * 2 = 200 mins Monthly operation = 200 * 22 = 75 hours Temporary Unstable Waste time!
18 Auto recover system • Classify failure report automatically • Re run test automation script when “Temporary unstable” Improve test automation operation
19 Auto recover system Test Result is Failure Predict reason Temporary unstable Re run Yes
20 Auto recover system Test Result is Failure Predict reason Temporary unstable Re run Yes Test result output • Console error message by Ranorex • Error page screenshot
21 Auto recover system Test Result is Failure Predict reason Temporary unstable Re run Yes Use data • Console error message • Screenshot • Previous prediction data Use Technology (Python) • Tesseract-OCR • Deep Neural Network with Keras
22 Auto recover system Test Result is Failure Predict reason Temporary unstable Re run Yes Input test report Extract message from screenshot Text base prediction Image base prediction No result Output prediction In progress Feedback
23 Auto recover system Test Result is Failure Predict reason Temporary unstable Re run Yes • Reduce waste time operation • Not waste test resource and time
24 Auto recover system benefit 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Oct-19 Nov-19 Dec-19 Jan-20 Feb-20 Mar-20 auto recovery manual recovery Ex. Date : March-20 Temporary unstable Ratio : 75% Operation time without system: 100 hours 100 * 0.65 = 65 hours saved 65%
25 Summary • Test automation operation takes time and includes boring task • Let’s automate some of operation • Auto recover system is not perfect yet, keep improvement
20200630 Rakuten QA meetup #2 "Improve test automation operation"

More Related Content

PPTX
Test Automation Improvement by Machine Learning Jasst'21 Tokyo
Sadaaki Emura
 
PPTX
Test automation engineer
Sadaaki Emura
 
PPTX
Test automation lesson
Sadaaki Emura
 
PPTX
ISTQB Advanced Test Manager Training
HiraQureshi22
 
PPTX
ISTQB Foundation and Selenium Java Automation Testing
HiraQureshi22
 
PPTX
Test management struggles and challenges in SDLC
Fumikazu FUJIWARA
 
PPTX
5 Considerations When Adopting Automated Testing
Bhupesh Dahal
 
PDF
Wso2con test-automation
krishantha_samaraweera
 
Test Automation Improvement by Machine Learning Jasst'21 Tokyo
Sadaaki Emura
 
Test automation engineer
Sadaaki Emura
 
Test automation lesson
Sadaaki Emura
 
ISTQB Advanced Test Manager Training
HiraQureshi22
 
ISTQB Foundation and Selenium Java Automation Testing
HiraQureshi22
 
Test management struggles and challenges in SDLC
Fumikazu FUJIWARA
 
5 Considerations When Adopting Automated Testing
Bhupesh Dahal
 
Wso2con test-automation
krishantha_samaraweera
 

What's hot (20)

PPTX
Qa process 2012
ashokack
 
PPTX
Test Automation Architecture That Works by Bhupesh Dahal
QA or the Highway
 
PDF
QA Process Overview for Firefox OS 2014
Anthony Chung
 
PDF
The limits of unit testing by Craig Stuntz
QA or the Highway
 
PDF
Test Automation
nikos batsios
 
PDF
End-to-End Test Automation for Both Horizontal and Vertical Scale
Erdem YILDIRIM
 
PDF
Tutorial ranorex
radikalzen
 
PDF
Test pyramid
Poornima Durairaj
 
PDF
Testing automation in agile environment
Perfecto Mobile
 
PDF
Automation testing in Agile project
Hien Nguyen
 
PDF
Test Automation
rockoder
 
PPTX
Test Automation Architecture in Microservices
Alper Mermer
 
PDF
Managing Test Labs Without the Headaches
Imaginet
 
PPTX
5 top pain points of test automation
Mikalai Alimenkou
 
PDF
Need for automation testing
99tests
 
PDF
Testing in FrontEnd World by Nikita Galkin
Sigma Software
 
PPTX
DaKiRY_BAQ2016_QADay_Marta Firlej "Microsoft Test Manager tool – how can we u...
Dakiry
 
PDF
KeytorcTestTalks #11 - Onur Başkirt, Agile Test Management with Testrail
Keytorc Software Testing Services
 
PDF
Agile in planday
Hien Nguyen
 
PPTX
Test Automation failure analysis
Prashant Chaudhary
 
Qa process 2012
ashokack
 
Test Automation Architecture That Works by Bhupesh Dahal
QA or the Highway
 
QA Process Overview for Firefox OS 2014
Anthony Chung
 
The limits of unit testing by Craig Stuntz
QA or the Highway
 
Test Automation
nikos batsios
 
End-to-End Test Automation for Both Horizontal and Vertical Scale
Erdem YILDIRIM
 
Tutorial ranorex
radikalzen
 
Test pyramid
Poornima Durairaj
 
Testing automation in agile environment
Perfecto Mobile
 
Automation testing in Agile project
Hien Nguyen
 
Test Automation
rockoder
 
Test Automation Architecture in Microservices
Alper Mermer
 
Managing Test Labs Without the Headaches
Imaginet
 
5 top pain points of test automation
Mikalai Alimenkou
 
Need for automation testing
99tests
 
Testing in FrontEnd World by Nikita Galkin
Sigma Software
 
DaKiRY_BAQ2016_QADay_Marta Firlej "Microsoft Test Manager tool – how can we u...
Dakiry
 
KeytorcTestTalks #11 - Onur Başkirt, Agile Test Management with Testrail
Keytorc Software Testing Services
 
Agile in planday
Hien Nguyen
 
Test Automation failure analysis
Prashant Chaudhary
 
Ad

Similar to 20200630 Rakuten QA meetup #2 "Improve test automation operation" (20)

PDF
Improve test automation operation
Rakuten Group, Inc.
 
PDF
The Pothole of Automating Too Much
TechWell
 
PDF
Testingexperience14 06 11_beersma
Bernd Beersma
 
PPTX
Presentation - Test Automation in Digital Transformation - IITPSA SIGIST 2016042
Wayne Sinclair
 
PPTX
Best practices for test automation
David Tzemach
 
PPTX
How to make Automation an asset for Organization
anuvip
 
PDF
Why Automation Fails—in Theory and Practice
TechWell
 
PDF
Why Test Automation Fails
Ranorex
 
PDF
CIO Review QA Mentor
Ruslan Desyatnikov
 
PDF
Strategies For Enterprise Test Automation.pdf
kalichargn70th171
 
PPTX
Chapter 1 - Introduction and Objectives for Test Automation
Neeraj Kumar Singh
 
PPTX
20180828 QA meetup
Sadaaki Emura
 
PDF
Creating a successful continuous testing environment by Eran Kinsbruner
QA or the Highway
 
PDF
How to manage test automation complexity
Dana Aonofriesei
 
PPTX
Myths of Automation Testing
Abhishek Saxena
 
PDF
QA Automation testing - Cloud deployed workflow application
Zado Technologies
 
PDF
Test automation Anecdotes
Cristiano Caetano
 
PPTX
Maturing your path toward DevOps with Continuous Testing
Perfecto Mobile
 
PPTX
Use Automation to Assist -Not Replace- Manual Testing
SmartBear
 
PDF
Successful Test Automation: A Manager’s View
TechWell
 
Improve test automation operation
Rakuten Group, Inc.
 
The Pothole of Automating Too Much
TechWell
 
Testingexperience14 06 11_beersma
Bernd Beersma
 
Presentation - Test Automation in Digital Transformation - IITPSA SIGIST 2016042
Wayne Sinclair
 
Best practices for test automation
David Tzemach
 
How to make Automation an asset for Organization
anuvip
 
Why Automation Fails—in Theory and Practice
TechWell
 
Why Test Automation Fails
Ranorex
 
CIO Review QA Mentor
Ruslan Desyatnikov
 
Strategies For Enterprise Test Automation.pdf
kalichargn70th171
 
Chapter 1 - Introduction and Objectives for Test Automation
Neeraj Kumar Singh
 
20180828 QA meetup
Sadaaki Emura
 
Creating a successful continuous testing environment by Eran Kinsbruner
QA or the Highway
 
How to manage test automation complexity
Dana Aonofriesei
 
Myths of Automation Testing
Abhishek Saxena
 
QA Automation testing - Cloud deployed workflow application
Zado Technologies
 
Test automation Anecdotes
Cristiano Caetano
 
Maturing your path toward DevOps with Continuous Testing
Perfecto Mobile
 
Use Automation to Assist -Not Replace- Manual Testing
SmartBear
 
Successful Test Automation: A Manager’s View
TechWell
 
Ad

More from Sadaaki Emura (14)

PPTX
Test Automation Journey 2023 useful knowledge to start test automation
Sadaaki Emura
 
PPTX
What is Unit Testing
Sadaaki Emura
 
PPTX
20220527_JaSST'22 Tohoku
Sadaaki Emura
 
PPTX
How to introduce test automation in VeriServe Test Automation Talk #2
Sadaaki Emura
 
PPTX
20211221 jasst nano_test automation operation
Sadaaki Emura
 
PPTX
20191029 automation struggle
Sadaaki Emura
 
PPTX
basic of Test automation installation
Sadaaki Emura
 
PPTX
20190531 jasst19 tohoku
Sadaaki Emura
 
PPTX
Struggles and Challenges in STLC in Ques No.13
Sadaaki Emura
 
PPTX
20190424 q ameetup-m -publish
Sadaaki Emura
 
PPTX
20181211 QA meetup in office
Sadaaki Emura
 
PPTX
JaSST'18 Hokkaido Improve Automation Testing
Sadaaki Emura
 
PPTX
Istqb : Test automation Engineer
Sadaaki Emura
 
PDF
QA improvement
Sadaaki Emura
 
Test Automation Journey 2023 useful knowledge to start test automation
Sadaaki Emura
 
What is Unit Testing
Sadaaki Emura
 
20220527_JaSST'22 Tohoku
Sadaaki Emura
 
How to introduce test automation in VeriServe Test Automation Talk #2
Sadaaki Emura
 
20211221 jasst nano_test automation operation
Sadaaki Emura
 
20191029 automation struggle
Sadaaki Emura
 
basic of Test automation installation
Sadaaki Emura
 
20190531 jasst19 tohoku
Sadaaki Emura
 
Struggles and Challenges in STLC in Ques No.13
Sadaaki Emura
 
20190424 q ameetup-m -publish
Sadaaki Emura
 
20181211 QA meetup in office
Sadaaki Emura
 
JaSST'18 Hokkaido Improve Automation Testing
Sadaaki Emura
 
Istqb : Test automation Engineer
Sadaaki Emura
 
QA improvement
Sadaaki Emura
 

Recently uploaded (20)

PPTX
Recipes for Real Time Voice AI WebRTC, SLMs and Open Source Software.pptx
Alberto González Trastoy
 
PPTX
UNIT 4 Total Quality Management .pptx
gokuld13012005
 
PPTX
Lecture Notes Electrical Wiring System Components
eedgecbhojpur
 
PPTX
Geodesy 1.pptx...............................................
abhi1361yadav
 
PDF
Automation-in-Manufacturing-Chapter-Introduction.pdf
pcmth867
 
PPTX
IOT PPTs Week 10 Lecture Material.pptx of NPTEL Smart Cities contd
ssusera400e8
 
PPTX
Engineering Ethics, Safety and Environment [Autosaved] (1).pptx
MehrabTaseenTalukder
 
PPTX
Infosys Presentation by1.Riyan Bagwan 2.Samadhan Naiknavare 3.Gaurav Shinde 4...
bapushinde7218
 
PDF
Well-logging-methods_new................
ZafriFarid
 
PPTX
Foundation to blockchain - A guide to Blockchain Tech
Davies Chacko
 
PDF
Evaluating the Democratization of the Turkish Armed Forces from a Normative P...
jpsjournal1
 
PDF
Operating System & Kernel Study Guide-1 - converted.pdf
mranoninvisib16
 
PPTX
KTU 2019 -S7-MCN 401 MODULE 2-VINAY.pptx
VinayB68
 
PPTX
MET 305 2019 SCHEME MODULE 2 COMPLETE.pptx
VinayB68
 
PPTX
M Tech Sem 1 Civil Engineering Environmental Sciences.pptx
ShriNRPrasad
 
PDF
Model Code of Practice - Construction Work - 21102022 .pdf
AinieButt1
 
PPT
CRASH COURSE IN ALTERNATIVE PLUMBING CLASS
rene militante
 
PDF
TFEC-4-2020-Design-Guide-for-Timber-Roof-Trusses.pdf
AinieButt1
 
PDF
Digital Logic Computer Design lecture notes
CHINNASUNKAIAH1
 
PPTX
bas. eng. economics group 4 presentation 1.pptx
kiribakimoses1993
 
Recipes for Real Time Voice AI WebRTC, SLMs and Open Source Software.pptx
Alberto González Trastoy
 
UNIT 4 Total Quality Management .pptx
gokuld13012005
 
Lecture Notes Electrical Wiring System Components
eedgecbhojpur
 
Geodesy 1.pptx...............................................
abhi1361yadav
 
Automation-in-Manufacturing-Chapter-Introduction.pdf
pcmth867
 
IOT PPTs Week 10 Lecture Material.pptx of NPTEL Smart Cities contd
ssusera400e8
 
Engineering Ethics, Safety and Environment [Autosaved] (1).pptx
MehrabTaseenTalukder
 
Infosys Presentation by1.Riyan Bagwan 2.Samadhan Naiknavare 3.Gaurav Shinde 4...
bapushinde7218
 
Well-logging-methods_new................
ZafriFarid
 
Foundation to blockchain - A guide to Blockchain Tech
Davies Chacko
 
Evaluating the Democratization of the Turkish Armed Forces from a Normative P...
jpsjournal1
 
Operating System & Kernel Study Guide-1 - converted.pdf
mranoninvisib16
 
KTU 2019 -S7-MCN 401 MODULE 2-VINAY.pptx
VinayB68
 
MET 305 2019 SCHEME MODULE 2 COMPLETE.pptx
VinayB68
 
M Tech Sem 1 Civil Engineering Environmental Sciences.pptx
ShriNRPrasad
 
Model Code of Practice - Construction Work - 21102022 .pdf
AinieButt1
 
CRASH COURSE IN ALTERNATIVE PLUMBING CLASS
rene militante
 
TFEC-4-2020-Design-Guide-for-Timber-Roof-Trusses.pdf
AinieButt1
 
Digital Logic Computer Design lecture notes
CHINNASUNKAIAH1
 
bas. eng. economics group 4 presentation 1.pptx
kiribakimoses1993
 

20200630 Rakuten QA meetup #2 "Improve test automation operation"

  • 1. Improve test automation operation June 30th, 2020 Sadaaki, Emura Leisure Product Dept. Rakuten, Inc.
  • 2. 2 Who am I Name : Sadaaki Emura (nickname M) Join in Rakuten : 2007 Career : Embedded engineer (2000-2002) Web engineer (2002-2015) Product manager (2015-2016) QA (2016~) Role: Test Automation Engineer Lead Birthplace : Kanazawa-city Hobby: jog , climbing , horse racing
  • 3. 3 Organization Developer group QA group Manual test team Test automation team @IllustAC ココ の人 :5 services
  • 5. 5 Today story 1. Test Automation operation is struggling 2. Improve test automation operation
  • 6. 6 Test automation brings benefits in proportion to coverage! • Reduce total cost of regression test • Quick feedback • Take work easy © FREEIMAGES
  • 7. 7 Test automation brings struggle actually ©photoAC focus • Many failure reports come • Synchronize latest specification • a few test automation engineers
  • 8. 8 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable Failure report classif y
  • 9. 9 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable • Good activity ! • Happy to report engineer! ©いらすとや
  • 10. 10 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable • Script has bug • Script covers old specification • Scenario dependency ©いらすとや
  • 11. 11 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable • Test data invalid • Application is not ready ©いらすとや
  • 12. 12 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable • Network disconnected • 500 error (releasing operation?) • 503 error (what’s happened?) ©いらすとや
  • 13. 13 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable
  • 14. 14 Classify failure report • Find bug • Test automation script issue • Environment issue • Temporary unstable 0… 5.3 21.1 73.4 Fing bug Test automation script issue Environment issue Temporary unstable
  • 15. 15 0… 5.3 21.1 73.4 Operation for Temporary unstable 1.Investigate failure report (common) 2.Run test automation script again 3.Check if result is success Ex. Operation Cost 1 operation = 2 mins Daily operation = 100 * 2 = 200 mins Monthly operation = 200 * 22 = 75 hours Temporary Unstable Waste time!
  • 16. 16 Today story 1. Test Automation operation is struggling 2. Improve test automation operation
  • 17. 17 0… 5.3 21.1 73.4 Operation for Temporary unstable 1.Investigate failure report (common) 2.Run test automation script again 3.Check if result is success Ex. Operation Cost 1 operation = 2 mins Daily operation = 100 * 2 = 200 mins Monthly operation = 200 * 22 = 75 hours Temporary Unstable Waste time!
  • 18. 18 Auto recover system • Classify failure report automatically • Re run test automation script when “Temporary unstable” Improve test automation operation
  • 19. 19 Auto recover system Test Result is Failure Predict reason Temporary unstable Re run Yes
  • 20. 20 Auto recover system Test Result is Failure Predict reason Temporary unstable Re run Yes Test result output • Console error message by Ranorex • Error page screenshot
  • 21. 21 Auto recover system Test Result is Failure Predict reason Temporary unstable Re run Yes Use data • Console error message • Screenshot • Previous prediction data Use Technology (Python) • Tesseract-OCR • Deep Neural Network with Keras
  • 22. 22 Auto recover system Test Result is Failure Predict reason Temporary unstable Re run Yes Input test report Extract message from screenshot Text base prediction Image base prediction No result Output prediction In progress Feedback
  • 23. 23 Auto recover system Test Result is Failure Predict reason Temporary unstable Re run Yes • Reduce waste time operation • Not waste test resource and time
  • 24. 24 Auto recover system benefit 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Oct-19 Nov-19 Dec-19 Jan-20 Feb-20 Mar-20 auto recovery manual recovery Ex. Date : March-20 Temporary unstable Ratio : 75% Operation time without system: 100 hours 100 * 0.65 = 65 hours saved 65%
  • 25. 25 Summary • Test automation operation takes time and includes boring task • Let’s automate some of operation • Auto recover system is not perfect yet, keep improvement

Editor's Notes

  • #3: これまでのキャリア 　最初、組み込み系のソフトウェアエンジニア　２年間 　その後、WEBエンジニア　１３年間 　次にプロダクトマネージャの経験を経る 　2016年、いまのSQA部署の構築から携わり、テスト自動化を合わせて構築しました（３年目）
  • #4: ・テスト環境は開発側が提供 ・QAはユーザー目線のテスト
  • #7: テスト自動化を毎日実行し、レグレッションテストで問題がないかを確認
  • #15: 元ネタ 2020/3-5 UNSTABLE 14844 ENVIRONMENT 4263 SCRIPT ISSUE 1068 BUG 38