Mika Mäntylä Beyond_Grepping_out - Testing Assembly 2022.pdf

Beyond Grepping - Using
AI in Pinpointing Anomaly
Events in Stability Testing
Prof. Mika Mäntylä, University of Oulu
PhD Martin Varela, Profilence
Contributions by
Post-Doc Yuqing Wang
Doctoral Student Shayan Hashemi
Docotoral Student Jesse Nyyssölä

Test automation needs to go
beyond pass / fail

Operational failures need
to go beyond pass/fail!
• Emergence of self-driving cars, farm
automation, and other software-
controlled non-human actors.
– Malfunctions in such devices could
result in life-threatening situations.
• Online services, assumptions that
software is always available
– Users do not care if poor software
service is due to operations (Ops) or
development (Dev)
This Photo by Unknown Author is licensed under CC BY

Log File – 65k lines
Log line
20499626 24.07.2020 03:31:58.243 Events 8416 8416 LogcatInfo auditd type=1400 audit(0.0:100719): avc: denied { read } for com
20499627 24.07.2020 03:31:58.243 Main 8416 8416 LogcatWarning cat type=1400 audit(0.0:100719): avc: denied { read } for nam
20499628 24.07.2020 03:31:58.355 Main 1013 1013 LogcatInfo android.hardware.health@2.0-service.crosshatch Entry state_: 1
20499629 24.07.2020 03:31:58.355 Main 1013 1013 LogcatInfo android.hardware.health@2.0-service.crosshatch Exit state_: 1 b
20499630 24.07.2020 03:31:58.363 System 10570 11459 LogcatDebug PowerUI can't show warning due to - plugged: true status un
20499631 24.07.2020 03:31:59.102 Main 1018 1090 LogcatDebug VSC @ 92597.370: [WO] tilt angle 87
20499632 24.07.2020 03:31:59.170 Main 24613 4314 LogcatInfo Icing Indexing com.android.chrome-internal.3p:WebPage from c
20499633 24.07.2020 03:31:59.205 Events 1745 2460 LogcatInfo am_compact [4190,com.google.process.gservices,all,37024,218
20499636 24.07.2020 03:31:59.743 Main 8463 8463 LogcatInfo Profilence btime 1595458124 92595.76 549346.36
20499639 24.07.2020 03:32:00.486 Main 24358 8505 LogcatDebug UiDaemon ScreenCaster stop
20499640 24.07.2020 03:32:00.490 Main 24358 8530 LogcatVerbose UiDaemon Invoke: {"MethodName":"setTracingLevel","ClassN

Log File – Part of 65k lines of single test log
Log line
How to find a root-cause?
Stardard industry practice:
Domain knowledge of system
and it logs + grepping

grep ”UIDaemon Screencaster stop”
Log line

grep ”UIDaemon Screencaster stop”
Log line
Might work if you have a good
hunch of what is wrong.
Otherwise like finding a
needle in a haystack
This Photo by Unknown Author is licensed under CC BY-SA

Log Analysis – State of the practice
• One log file + Text editor Find / Grep

Compare: good log vs failure log
• Manually
• Using Tools (diff, KDiﬀ3, Beyond Compare)
Log line
20499626 24.07.2020 03:31:58.243 Events 8416 8416 LogcatInfo auditd type=1400 aud
20499627 24.07.2020 03:31:58.243 Main 8416 8416 LogcatWarning cat type=1400 audit
20499628 24.07.2020 03:31:58.355 Main 1013 1013 LogcatInfo android.hardware.hea
20499630 24.07.2020 03:31:58.363 System 10570 11459 LogcatDebug PowerUI can't sho
20499631 24.07.2020 03:31:59.102 Main 1018 1090 LogcatDebug VSC @ 92597.370: [W
20499632 24.07.2020 03:31:59.170 Main 24613 4314 LogcatInfo Icing Indexing com.an
20499633 24.07.2020 03:31:59.205 Events 1745 2460 LogcatInfo am_compact [4190,co
20499636 24.07.2020 03:31:59.743 Main 8463 8463 LogcatInfo Profilence btime 15954
20499639 24.07.2020 03:32:00.486 Main 24358 8505 LogcatDebug UiDaemon ScreenCa
20499640 24.07.2020 03:32:00.490 Main 24358 8530 LogcatVerbose UiDaemon Invoke:
Log line
20499626 24.07.2020 03:31:58.243 Events 8416 8416 LogcatInfo auditd type=1400 audit(0.0:100719): avc: denied { read } for comm="cat" name="diskstats" dev="proc" ino=4026532182 scontext=u:r:shell:s0 tcontext=u:object_r:
20499627 24.07.2020 03:31:58.243 Main 8416 8416 LogcatWarning cat type=1400 audit(0.0:100719): avc: denied { read } for name="diskstats" dev="proc" ino=4026532182 scontext=u:r:shell:s0 tcontext=u:object_r:proc_diskstats
20499628 24.07.2020 03:31:58.355 Main 1013 1013 LogcatInfo android.hardware.health@2.0-service.crosshatch Entry state_: 1 charger_status: Charging batteryLevel: 97
20499629 24.07.2020 03:31:58.355 Main 1013 1013 LogcatInfo android.hardware.health@2.0-service.crosshatch Exit state_: 1 batteryLevel: 100
20499630 24.07.2020 03:31:58.363 System 10570 11459 LogcatDebug PowerUI can't show warning due to - plugged: true status unknown: false
20499632 24.07.2020 03:31:59.170 Main 24613 4314 LogcatInfo Icing Indexing com.android.chrome-internal.3p:WebPage from com.android.chrome using seqno 0
20499633 24.07.2020 03:31:59.205 Events 1745 2460 LogcatInfo am_compact [4190,com.google.process.gservices,all,37024,21872,14904,23152,-3488,0,-3488,3488,45,1,92575207,700,15,801276,-3488]
20499640 24.07.2020 03:32:00.490 Main 24358 8530 LogcatVerbose UiDaemon Invoke: {"MethodName":"setTracingLevel","ClassName":"UiDaemon","Args":[16]}

Compare good log vs failure log
Log line
20499631 24.07.2020 03:31:59.102 Main 1018 1090 LogcatDebug VSC @ 92597.370: [W
20499632 24.07.2020 03:31:59.170 Main 24613 4314 LogcatInfo Icing Indexing com.an
20499633 24.07.2020 03:31:59.205 Events 1745 2460 LogcatInfo am_compact [4190,co
20499636 24.07.2020 03:31:59.743 Main 8463 8463 LogcatInfo Profilence btime 15954
20499640 24.07.2020 03:32:00.490 Main 24358 8530 LogcatVerbose UiDaemon Invoke:
Log line
Better – We are using
historical data
but with just one file
or max two filesfo

Log Analysis – State of the practice
• One log file approach with Text editor Find / Grep
• Two log file approach (good vs bad) with text comparison tools

Take all good logs and compare them to bad
Log line
20499626 24.07.2020 03:31:58.243 Events 8416 8416 LogcatInfo au
20499627 24.07.2020 03:31:58.243 Main 8416 8416 LogcatWarning
20499628 24.07.2020 03:31:58.355 Main 1013 1013 LogcatInfo an
20499630 24.07.2020 03:31:58.363 System 10570 11459 LogcatDebug
20499631 24.07.2020 03:31:59.102 Main 1018 1090 LogcatDebug V
20499632 24.07.2020 03:31:59.170 Main 24613 4314 LogcatInfo Ic
20499633 24.07.2020 03:31:59.205 Events 1745 2460 LogcatInfo am
20499636 24.07.2020 03:31:59.743 Main 8463 8463 LogcatInfo Pr
20499639 24.07.2020 03:32:00.486 Main 24358 8505 LogcatDebug
20499640 24.07.2020 03:32:00.490 Main 24358 8530 LogcatVerbose
Log line
Log line
20499626 24.07.2020 03:31:58.243 Events 8416 8416 LogcatInfo auditd type=1400 audit(0.0:100719): avc: denied { read } for comm="cat" name="diskstats" dev="proc" ino=4026532182 scontext=u:r:shell:s0 tcontext=u:object
20499627 24.07.2020 03:31:58.243 Main 8416 8416 LogcatWarning cat type=1400 audit(0.0:100719): avc: denied { read } for name="diskstats" dev="proc" ino=4026532182 scontext=u:r:shell:s0 tcontext=u:object_r:proc_diskst
Log line
20499626 24.07.2020 03:31:58.243 Events 8416 8416 LogcatInfo auditd type=1400 audit(0.0:100719): avc: denied { read } for comm="cat" name="diskstats" dev="proc" ino=4026532182 scontext=u:r:shell:s0 tcontext=u:obj
20499627 24.07.2020 03:31:58.243 Main 8416 8416 LogcatWarning cat type=1400 audit(0.0:100719): avc: denied { read } for name="diskstats" dev="proc" ino=4026532182 scontext=u:r:shell:s0 tcontext=u:object_r:proc_dis
Log line
20499626 24.07.2020 03:31:58.243 Events 8416 8416 LogcatInfo auditd type=1400 audit(0.0:100719): avc: denied { read } for comm="cat" name="diskstats" dev="proc" ino=4026532182 scontext=u:r:shell:s0 tcontext=u:
20499627 24.07.2020 03:31:58.243 Main 8416 8416 LogcatWarning cat type=1400 audit(0.0:100719): avc: denied { read } for name="diskstats" dev="proc" ino=4026532182 scontext=u:r:shell:s0 tcontext=u:object_r:proc_
Log line
20499626 24.07.2020 03:31:58.243 Events 8416 8416 LogcatInfo auditd type=1400 audit(0.0:100719): avc: denied { read } for comm="cat" name="diskstats" dev="proc" ino=4026532182 scontext=u:r:shell:s0 tcontext=
20499627 24.07.2020 03:31:58.243 Main 8416 8416 LogcatWarning cat type=1400 audit(0.0:100719): avc: denied { read } for name="diskstats" dev="proc" ino=4026532182 scontext=u:r:shell:s0 tcontext=u:object_r:pr
Log line
20499626 24.07.2020 03:31:58.243 Events 8416 8416 LogcatInfo auditd type=1400 audit(0.0:100719): avc: denied { read } for comm="cat" name="diskstats" dev="proc" ino=4026532182 scontext=u:r:shell:s0 tconte
20499627 24.07.2020 03:31:58.243 Main 8416 8416 LogcatWarning cat type=1400 audit(0.0:100719): avc: denied { read } for name="diskstats" dev="proc" ino=4026532182 scontext=u:r:shell:s0 tcontext=u:object_r
Log line
20499626 24.07.2020 03:31:58.243 Events 8416 8416 LogcatInfo auditd type=1400 audit(0.0:100719): avc: denied { read } for comm="cat" name="diskstats" dev="proc" ino=4026532182 scontext=u:r:shell:s0 tco
20499627 24.07.2020 03:31:58.243 Main 8416 8416 LogcatWarning cat type=1400 audit(0.0:100719): avc: denied { read } for name="diskstats" dev="proc" ino=4026532182 scontext=u:r:shell:s0 tcontext=u:objec

How can we compare a bad log against all good logs?
• Log file = sequence of events:
– e_start; e_do_stuff_1; e_do_stuff_2; e_failure;
e_end;
Log line
20499626 24.07.2020 03:31:58.243 Events 8416 8416 LogcatInfo auditd type=1400 audit(0.0:100719): avc: denied { read } for comm="cat" name="diskstats" dev="proc" ino=4026532182 scontext=u:r:shel
20499627 24.07.2020 03:31:58.243 Main 8416 8416 LogcatWarning cat type=1400 audit(0.0:100719): avc: denied { read } for name="diskstats" dev="proc" ino=4026532182 scontext=u:r:shell:s0 tcontext=
Log Parsing

– e_start; e_do_stuff; e_failure; e_end;
• Bioinformatics: Gene = sequence of proteins
– Task: Gene sequence alignment
– Shows differences in genes
– Compare many against one

– e_start; e_do_stuff; e_failure; e_end;
• Bioinformatics: Gene = sequence of proteins
– Task: Gene sequence alignment
– Shows differences in genes
– Compare many against one
Anomaly
Normal 1
Normal 2
Normal 3
Normal 4

– e_start; e_do_stuff;
e_failure; e_end;
• Natural language processing:
Text = sequence of words
– Task: Masked / Next word
prediction
– Use all of the history to predict
– Assign probability to masked
event
• e_start; e_do_stuff; ?;
e_end;

Computers are excellent in remembering everything
Log line
20499631 24.07.2020 03:31:59.102 Main 1018 1090 LogcatDebug V
20499632 24.07.2020 03:31:59.170 Main 24613 4314 LogcatInfo Ic
20499633 24.07.2020 03:31:59.205 Events 1745 2460 LogcatInfo am
20499636 24.07.2020 03:31:59.743 Main 8463 8463 LogcatInfo Pr
20499640 24.07.2020 03:32:00.490 Main 24358 8530 LogcatVerbose
Log line
Log line
Log line
Log line
Log line
Log line
Log line

Log Analysis – State of the practice and state of the art
• One log file approach with Text editor, Grep
• Two log file approach (good vs bad) with text comparison tools
• State of the art – Compare all good against bad
– Research in bioinformatics and NLP can help

Parsing – From Log Lines to
Event Streams
• Typically considers Log Message
• Parsing = specialized text clustering

After parsing – Events A0001 A0002

After parsing – Event A0001 A0002 A0003

Example - Scoring log
events for suspiciousness
• A sequence of log events E1 E2 E4
• E4 is suspicious if E4 rarely appears after E1 E2
• Data
• event sequence E1 E2 exists 100 times
• event sequence E1 E2 E4 exists 10 times,
• Suspiciousness Scores
• Occurrence count for the event E4 is 10
• Probability for E4 is 0.1 (10/100)
https://arxiv.org/abs/2202.09214

Anomaly scores
Count: 85
for sequence SoS SoS SoS SoS A0001

Anomaly scores (window size = 5)
Count: 85
for sequence SoS SoS SoS A0001 A0002

Anomaly scores (window size = 5)
Count: 2
for sequence SoS SoS A0001 A0002 A0003

Anomaly scores
Probability: 0.31 (85 / 273)
for sequence SoS SoS SoS SoS A0001

Anomaly scores
Probability: 1.00 (85 / 85)
for sequence SoS SoS SoS A0001 A0002

Anomaly scores
Probability: 0.02 (2 / 85)
for sequence SoS SoS A0001 A0002 A0003

Event sequence with > 65k events https://arxiv.org/abs/2202.09214

AI model comparison – Event prediction accuracy
• CNN or LSTM (deep learning) produce nearly identical results
• Larger window size can lead in degrading performance in N-Gram (classical computing)
• Window size = how many events are used for prediction

Computational efficiency – Ngram is faster
• ~1000 times faster in training in “Office PC” (13s vs. 3h 50min)
• ~9 times faster in inference “GPU PC” (7s vs. 64s)
• N-Gram is like a memory
• Model training: store all n-grams of given lengths, their counts, and predictions.
• Model usage: accessing the data that is stored
• Implemented with Python Dictionaries that have O(1) lookup time
• Deep-learning (LSTM/CNN) is a computational model
• Model training: train neural network from the data
• Model usage: push a given n-gram through the trained neural network to get the
prediction.

Computational efficiency at Company
• N-Gram better for anomaly detection in production
• The numbers are for 1 test case.
• Each product tens or hundreds test cases
• Several products under test
• Multiple all execution times by 1000 for the whole company!

Observing low scores to find anomalies

Summary - How to pinpoint problem areas in logs?
• State of the practice
• 1 file approach (bad file only) -> grepping and text editors
• 2 file approach (good vs bad) -> comparisons & diffing
• State of the art
• Multi file approach (all good vs bad) -> log line
suspiciousness scoring
• Classical computing superior computational efficiency
• Deep learning slightly better accuracy
Plan Code Build Test
Release
Operate Deploy
Monitor

Interested in helping
with research and using
online log analyzer?
• Leave a business card
• Email me: mika.mantyla@oulu.fi
title: “Online log analyzer”

References
• Mäntylä, M., Varela, M., & Hashemi, S. (2022, April). Pinpointing Anomaly
Events in Logs from Stability Testing–N-Grams vs. Deep-Learning. In 2022
IEEE International Conference on Software Testing, Verification and
Validation Workshops (ICSTW) (pp. 285-292). IEEE.
https://arxiv.org/pdf/2202.09214.pdf
• Wang, Y., Mäntylä, M. V., Liu, Z., Markkula, J., & Raulamo‐jurvanen, P.
(2022). Improving test automation maturity: A multivocal literature review.
Software Testing, Verification and Reliability, 32(3), e1804.
https://onlinelibrary.wiley.com/doi/full/10.1002/stvr.1804
• Nyyssölä, J., Mäntylä, M., & Varela, M. (2022). How to Configure Masked
Event Anomaly Detection on Software Logs?. International Conference on
Software Maitenance and Evolution (ICSME 2022).

Mika Mäntylä Beyond_Grepping_out - Testing Assembly 2022.pdf

More Related Content

Recently uploaded (20)

Featured (20)

Mika Mäntylä Beyond_Grepping_out - Testing Assembly 2022.pdf