Test Automation Improvement by Machine Learning Jasst'21 Tokyo

Editor's Notes

• #2: Hello! Thank you for joining my presentation. I’m Emura Sadaaki from Rakuten. Today, I give you a presentation about “Test Automation Improvement by Machine Learning” Before start presentation, let me introduce myself.
• #3: Go back to presentation. Today , this presentation is 2 topics. 1st topic is “Test automation has daily operation and issue” 2nd topic is “This issue is resolved by system with machine learning” Let’s get started
• #4: Agenda is here. 1st : Our QA team background 2nd : What is Test automation operation in our team and this issue 3rd : investigate this operation issue in detail 4th : improve operation to resolve this issue Finally : conclusion
• #5: Let go 1st agenda “Background”
• #6: Our organization is this. Our dept. have mainly 2 type group. Left side is developer group to create product. This group has product manager and engineer. There are 11 developer group in our Dept. Right side is QA group to check product quality when every application release. This group has manual test team and test automation team. I belong to test automation team. This QA group do test for 11 developer group services.
• #7: 11 services is here.
• #8: I explain test automation team responsibility in QA group. Test automation team has 3 responsibility. 1st : test automation covers regression . 2nd : update regression script and add new script during project. 3rd : do regression test in test environment everyday. I explain each responsibility in detail one by one.
• #9: 1st responsibility is test automation covers regression. This image show all function in product . Blue area is regression area. Test automation cover this blue area. Let’s say “one new project start”. In this time, Red area appears. This is new specification. And it is new project scope. Manual test focus on this project scope , it’s red area. Automation test cover regression during this project, it’s blue area.
• #10: 2nd responsibility is “update regression script and create new script during project”. As I said in 1st responsibility, test automation focus on regression area. Area No.1, it is regression and this specification is changed to satisfy new specification. So test automation basically does not work well so to support last specification , we should modify script . Other side, Area No.2 , this function is totally new . Test automation does not cover this area yet. So to support this new function, we should create new script then add to existing regression. Finally we expand regression area
• #11: 3rd responsibility is Regression test in test environment everyday. Not only testing during project, All test automation regression is scheduled everyday then check quality every time . This main objective is finding bug. others is finding environment issue, release operation rehearsal issue, non-functional bug (in Japanese 非機能テスト) Etc..
• #12: As above, Test automation responsibility are - cover regression - maintain script - run regression test everyday To keep these responsibility , many operations come up!
• #13: In 1st agenda I introduced our QA team organization and test automation responsibility. In next agenda, I explain test automation operation and issue during this operation.
• #14: 3rd responsibility is “All regression test is scheduled everyday.” In this regression test, we have 3 operations 1st check test automation results 2nd investigate test result if test automation failed 3rd handle test automation results I explain 3 operation one by one
• #15: 1st Operation . Check test automation results. Our team check if all regression test work well or not everyday. Currently we have around 20 Jenkins servers. In these Jenkins, around 2,000 regression test exist and run everyday. To check this so many test status, we have dashboard right image, this provide the test status summary in each service. Ex. How many test is success , how many test is failed . If test failed, we get the test result from this dashboard.
• #16: 2nd operation is investigate test result of test automation failed. We get test result from dashboard, like this image. This report has information Where test is failed What test data is used Failed page Screenshot Etc. From test result , we investigate why test is failed. Failed reason is classified into mainly 4 category. 1st application bug 2nd test automation script bug 3rd environment issue 4th temporary unstable
• #17: 3rd operation , we handle test automation results. After investigate failed test, to be test success , we fix issue Let me explain how to fix issue one by one
• #18: 1st type “Application bug” In this case, test automation does not work till developer fix application bug. So 1st we report bug to engineers Engineers get this bug report, fix application bug then notice QA team . Finally run test automation again then check application bug fixed
• #19: 2nd type “Test automation script bug” In this case, it’s our side issue. So check specification again, fix test automation script bug by myself. Finally run test automation again then check test automation work well.
• #20: 3rd type “Environment issue” This type example is “maintenance mode” Developer set this mode to do some operation. During this maintenance, test environment does not work. So during this maintenance , we stop test automation. Communicate developer, get information maintenance is finished, retry test automation .then check “ test automation work well “. If we know this maintenance schedule in advance, manage test automation scheduler .
• #21: Final 4th type “temporary unstable” This situation is test environment does not work “in short period”. For example. Network connection difficulty . Application cause timeout. Web server cause busy In this case, fixing solution is very simple only “retry test automation “ Although we take much time to investigate this type issue, final operation is “RETRY” So this temporary unstable issue is very non productive and take much time.
• #22: To run All regression test everyday, Test automation has some operations. In these operations, temporary unstable issue is not productive and take much time.
• #23: 3rd agenda investigate this test automation issue in detail.
• #24: During test automation operation, temporary unstable issue operation is not productive and take much time. But this “much time” is hypothesis at that time. 1st , we need to verify this fact (take much time) . if true, consider improvement method to resolve it.
• #25: To verify this fact, we collect all test result and find out issue trend. So build this system. 1st we collect “all test results from 20 Jenkins servers”, and stock this information in DB automatically. We check these test result, then classify all test results by failed reason category in dashboard.
• #26: This is classify operation web tool . We investigate why test was failed, then classify failed reason category. choose failed reason from mainly 4 category.
• #27: This is the summary failed reason category. This circle is reason ratio Red is “application bug”. it’s 0.5 % Blue is “Test automation script bug” . It’s about 5% Yellow is “Environment issue“. It’s about 20 % Green is “Temporary unstable”. It’s three quarters ! This type issue occupy most of all
• #28: As I said , when this temporary unstable issue is happened, operation to fix issue is this step. 1st Investigate failed result 2nd retry test automation Finally check result. Lets calculate operation time. For example this 1 operation take 2 mins daily operation 300 In this condition, we take about 100 hours per month. It’s huge operation cost!
• #29: Next agenda is “improve operation”
• #30: By investigating issue, we found out “temporary unstable” type issue cause huge operation cost. Also this operation is non-productive . We decided to improve this operation to reduce human cost.
• #31: This is system to improve operations. Called “Auto healing system” 1st we use classified data during investigate issue trend as training data (in Japanese 教師データ) 2nd This system predict failed reason by machine learning with this training data. Finally , If there is a temporary unstable issue, retry test automation in Jenkins
• #32: This is flow chart
• #33: 1st step : test executes and failed. In this time, we get test results , mainly 2 things - text error message - Error page screenshot
• #34: 2nd Step : predict failed reason To predict it, Use test result and training data. This system use machine learning technology. Mainly 2 tech. -Tessact OCR -Deep Neural Network with Keras-library
• #35: Prediction step detail is here
• #36: As I said , we use test results “Test error message and Screenshot” as system input ,
• #37: We extract text error message from screenshot. Example. Invalid operation,不正な操作が行われました in Japanese from screenshot
• #38: Using both text error messages. 1st prediction is done with training data.
• #39: If 1st prediction is difficult, 2nd prediction start , image base prediction.
• #40: After prediction, this prediction result feedback to training data
• #41: 3rd step check predict result , decide to retry test automation When Prediction is “temporary unstable”, retry test automation. Otherwise do nothing.
• #42: Auto healing do “Prediction” and “RED AREA” - “Retry test automation” in auto.
• #43: Auto healing system benefit is here 1st benefit : Retry operation is dismissed . Then human cost is ZEO. 2nd benefit : This auto healing run every time after test automation run. Some failed test is fixed soon in auto. So test results keep stable 3rd benefit : Retry operation is done only when failed test might be fixed . That’s why we can make good use of test environment resources.
• #44: This is actual output This graph mean 100 % is all failed test automation. RED is fixed by manual . BLUE is fixed by auto healing, this fix operation is ZERO human operation Pick up July case. For example,monthly operation take about 112 hours. 78% is fixed by auto healing. So ,about 88 hours human cost is reduct! It’s very huge outcome.
• #45: Conclusion
• #46: I explained test automation responsibility. To keep this responsibility, we have many test automation operations, and some of them are non productive activity. I introduce Auto healing system with Machine learning. It can reduce these operation. We can reduce non productive cost, but this auto healing is not perfect yet. For example, prediction accuracy is not high, feedback is manual operation. That why there is still room for improvement auto healing system. So we want to improve this system more and more.
• #47: That it from me. Thank you for attending my presentation!