![Киев 2017
Risk-Based implementation: Risk Identification
Risk based testing in reality: How to make it works on your project?
[-] Poor requirements
[-] Lots of changes
[-] No past experience
[-] Busy Stakeholders
[-] No Trust
Goal: Quality Risks List
New project
Identify + Communicate
Present
Apply
Learn DOMAIN (!)
Google
Brainstorming
Establish standartized KB
Ask RIGHT questions in RIGHT place
Focus on YOUR Team affected Area
Identify + Establish + Communicated
[+] Business availability
[+] No defined Processes
[+] Time](https://image.slidesharecdn.com/v-170927203546/85/QA-Fest-2017-9-320.jpg)
![Киев 2017
Risk-Based implementation: Risk Identification
Risk based testing in reality: How to make it works on your project?
Goal: Quality Risks List
Ongoing project
Find ‘the RIGHT Guy’
Investigate
Brainstorm and Identify
bottlenecks
Identify Risks
Prepare to present
[-] Low Business availability
[-] Documentation: poor/surplus
[-] Custom process
[+] Trust
[+] Domain expertise
[+] Available past experience
Define Implementation Plan
Present Solution
Prepare to implement
[?] Existed test cases](https://image.slidesharecdn.com/v-170927203546/85/QA-Fest-2017-10-320.jpg)
![Киев 2017
Risk-Based implementation: Risk Assessment
Risk based testing in reality: How to make it works on your project?
[-] Not clear requirements
[-] Not fully defined expectation
[-] No past experience
Goal: assign Risk Priority Number (RPN)
New project
[+] Right people available
Provide presentation
Make sure that all understand WHAT Likelihood
and Impact mean
All understand the Measurement Scale
Explain the RULES
Execute Assessment
RIGHT people involved
Appropriate time allocated
Establish Plan how to deal with ‘open questions’](https://image.slidesharecdn.com/v-170927203546/85/QA-Fest-2017-13-320.jpg)
![Киев 2017
Risk-Based implementation: Risk Assessment
Risk based testing in reality: How to make it works on your project?
Goal: assign Risk Priority Number (RPN)
Ongoing project
[-] Formed Standards of Priority Scale
[+] Available past experience
Analyse Past Project Expirience
Execute Assessment
People involved (!)
Appropriate time allocated
Establish Plan how to deal with ‘open
questions’
Provide presentation
Make sure that all understand WHAT Likelihood
and Impact mean
All understand the Measurement Scale
Explain the RULES](https://image.slidesharecdn.com/v-170927203546/85/QA-Fest-2017-14-320.jpg)
![Киев 2017
Risk-Based implementation: Risk Mitigation
Risk based testing in reality: How to make it works on your project?
[1-5] Extensive Run ~ 50 % from total test design, implementation end execution effort
a large number of tests that are both broad and deep, exercising combinations and variations of
interesting conditions.
[6-10] Broad Run ~ 25-35 % from total test design implementation end execution effort
a medium number of tests that exercise many different interesting conditions
[11-15] Cursory Run ~ 5-15 % from total test design implementation end execution effort
a small number of tests that sample the most interesting conditions
[16-20] Opportunity ~ 5 % from total test design implementation end execution effort
Leverage other tests or activities to run a test or two of an interesting condition (small investment of time
and effort)
[21-25] Report bugs
MAKE sure that EVERY Stakeholder understand definitions](https://image.slidesharecdn.com/v-170927203546/85/QA-Fest-2017-16-320.jpg)
![Киев 2017
Risk-Based implementation: Risk Mitigation
Risk based testing in reality: How to make it works on your project?
RESULT = Extend of testing assigned
1. System response > 20 ms under full load
(performance)
2. Patches and releases won’t install correctly
(supportability)
3. Incorrect exchange rate for currency pair
GBP:CNY (functional)
4. User failed to log in and authentication failed
(functional)
5. User find interface too complex (usability)
RPN
15
4
2
8
20
Extend of
testing
C
E
E
B
O
TCs
Suite-15
Suite-18
Suite-1
Suite-2
CR-5
US-101
TT-10
US-5
US-65
NA
US
Issues
I-5, I-9
I-15
I-25
I-65
I-8
TRACING
yes
no
yes
yes
yes
Is
Aut.
Extend of testing
1-5 = [E] = Extensive
6-10 = [B] = Broad
11-15 = [C] = Cursory
15-20 = [O] = Opportunity
21-25 = [RP] = Report Bugs](https://image.slidesharecdn.com/v-170927203546/85/QA-Fest-2017-17-320.jpg)
![Киев 2017
Risk-Based implementation: Risk Mitigation
Risk based testing in reality: How to make it works on your project?
Goal: allocate test effort
[-] Mixed coverage
[-] Catalogue Maintenance problems
[-] Miss issues (RB, O)
[+] Keep test FOCUS on WHAT is METTER
[+] Limit test afford with RNP
1 test case covers ONLY 1 entity
Automate Risk Catalog
Establish Team Rules
Allocate time for Experience based testing
New project & Ongoing Project
How to build Report with JIRA: Atlassian Support/JIRA Software Cloud/JIRA Knowledge Base
Atlassian Marketplace: qTest for JIRA Test Management](https://image.slidesharecdn.com/v-170927203546/85/QA-Fest-2017-18-320.jpg)
![Киев 2017
Risk-Based implementation: Risk Management
Risk based testing in reality: How to make it works on your project?
Goal: analyze, monitor and control
[-] Catalogue Maintenance
[-] Process Maintenance
[+] Time Constraint problem Resolved
(breadth-first and depth-first)
[+] Bug-fix efforts focused on high-risk
problems
[+] Now we known the Scope
Use Retro
Build and Analyze Test Report based on
Risks
Communicate the process
New project & Ongoing Project](https://image.slidesharecdn.com/v-170927203546/85/QA-Fest-2017-19-320.jpg)
QA Fest 2017. Ирина Жилинская. Тестирование основанное на рисках в реальности: как с ним работать
- 1. Киев 2017Risk based testing in reality: How to make it works on your project? Risk based testing in reality: How to make it works on your project?
- 2. Киев 2017 Speaker Info • 7 years in IT in roles BA & QA: Financial Services Industry, Sport Industry • ISTQB Trainer Risk based testing in reality: How to make it works on your project? IRYNA ZHYLINSKA LEAD SOFTWARE TEST ENGINEER EPAM
- 3. Киев 2017 Agenda: is Risk-Based testing approach? values it brings? metrics it gives? WHAT HOW to implement Risk-based testing on New Project on existed (on-going Project) pros and cons risk based approach launching on your Project Risk based testing in reality: How to make it works on your project?
- 4. Киев 2017 WHAT: is Risk-Based testing approach? Risk based testing in reality: How to make it works on your project? KEY TERMINOLOGY POSSIBLE NEGATIVE RISK + = Can we test It? OUTCOME PRODUCT Quality PROJECT YES NO
- 5. Киев 2017 WHAT: is Risk-Based testing approach? Risk based testing in reality: How to make it works on your project? KEY TERMINOLOGY LIKELYHOOD IMPACT Risk Priority Number x = RBCS link Article: "Risk Based Testing: What It Is and How You Can Benefit" by Rex Black
- 6. Киев 2017 WHY: Risk Based approach values … for any system infinite # of tests exists …so you have to prioritize what is matter to test and you have to measure what you test but you DO NOT have forever to test Risk based testing in reality: How to make it works on your project?
- 7. Киев 2017 HOW: implement Risk-Based approach Risk based testing in reality: How to make it works on your project? New project On-going project Just started No history Ongoing > 2-3 years history Practical Examples:
- 8. Киев 2017 HOW: Risk-Based testing process looks like? Risk Identification Risk Assessment Risk Management Risk Mitigation Risk based testing in reality: How to make it works on your project? OUTPUT Quality RISKs List RPN Test Prioritization Correct Effort allocation Defined Scope STEP Execute Report Analyze Allocate new Risks Monitor and Control
- 9. Киев 2017 Risk-Based implementation: Risk Identification Risk based testing in reality: How to make it works on your project? [-] Poor requirements [-] Lots of changes [-] No past experience [-] Busy Stakeholders [-] No Trust Goal: Quality Risks List New project Identify + Communicate Present Apply Learn DOMAIN (!) Google Brainstorming Establish standartized KB Ask RIGHT questions in RIGHT place Focus on YOUR Team affected Area Identify + Establish + Communicated [+] Business availability [+] No defined Processes [+] Time
- 10. Киев 2017 Risk-Based implementation: Risk Identification Risk based testing in reality: How to make it works on your project? Goal: Quality Risks List Ongoing project Find ‘the RIGHT Guy’ Investigate Brainstorm and Identify bottlenecks Identify Risks Prepare to present [-] Low Business availability [-] Documentation: poor/surplus [-] Custom process [+] Trust [+] Domain expertise [+] Available past experience Define Implementation Plan Present Solution Prepare to implement [?] Existed test cases
- 11. Киев 2017 Risk-Based implementation: Risk Identification Risk based testing in reality: How to make it works on your project? Ongoing projectNew project Define Quality Risks Catalogue Maintenance Plan for Risks (Vision) Risks Reviews Define Implementation plan • Collect data from the past • Trace past data with identified Risks • Define missed area • Allocate time and Resourses READY TO MOVE TO RISK ASSESMENT Quality Risks Catalogue
- 12. Киев 2017 Risk-Based implementation: Risk Identification Risk based testing in reality: How to make it works on your project? RESULT = Risks Check List System response > 20 ms under full load Patches and releases won’t install correctly Incorrect exchange rate for currency pair GBP:CNY User failed to log in and authentication failed User find interface too complex Performance Supportability Functional Functional Usability US-101 TT-10 US-5 US-65 NA TC-15 TC-25 TC-35 TC-88 NA yes no yes yes NA User Story Test Case Is Aut. Note: configure your tools to pick all this data automatically Link for Templates: RBCS: Generic Quality Risks List and General Quality Risk Categories In case on-going project better to pick up existed information from the begining
- 13. Киев 2017 Risk-Based implementation: Risk Assessment Risk based testing in reality: How to make it works on your project? [-] Not clear requirements [-] Not fully defined expectation [-] No past experience Goal: assign Risk Priority Number (RPN) New project [+] Right people available Provide presentation Make sure that all understand WHAT Likelihood and Impact mean All understand the Measurement Scale Explain the RULES Execute Assessment RIGHT people involved Appropriate time allocated Establish Plan how to deal with ‘open questions’
- 14. Киев 2017 Risk-Based implementation: Risk Assessment Risk based testing in reality: How to make it works on your project? Goal: assign Risk Priority Number (RPN) Ongoing project [-] Formed Standards of Priority Scale [+] Available past experience Analyse Past Project Expirience Execute Assessment People involved (!) Appropriate time allocated Establish Plan how to deal with ‘open questions’ Provide presentation Make sure that all understand WHAT Likelihood and Impact mean All understand the Measurement Scale Explain the RULES
- 15. Киев 2017 Risk-Based implementation: Risk Assessment Risk based testing in reality: How to make it works on your project? RESULT = Risk Check List with RNP assessed 1. System response > 20 ms under full load (performance) 2. Patches and releases won’t install correctly (supportability) 3. Incorrect exchange rate for currency pair GBP:CNY (functional) 4. User failed to log in and authentication failed (functional) 5. User find interface too complex (usability) 3 4 2 4 4 Likelihood Impact Current RPN 5 1 1 2 5 15 4 2 8 20 Scale 1 =Critical 2 = Major 3 = Medium 4 = Low 5 = Very Low Past RNP 12-14 5 2-3 6 0 Analysed expirience -> form RNP based on existed data
- 16. Киев 2017 Risk-Based implementation: Risk Mitigation Risk based testing in reality: How to make it works on your project? [1-5] Extensive Run ~ 50 % from total test design, implementation end execution effort a large number of tests that are both broad and deep, exercising combinations and variations of interesting conditions. [6-10] Broad Run ~ 25-35 % from total test design implementation end execution effort a medium number of tests that exercise many different interesting conditions [11-15] Cursory Run ~ 5-15 % from total test design implementation end execution effort a small number of tests that sample the most interesting conditions [16-20] Opportunity ~ 5 % from total test design implementation end execution effort Leverage other tests or activities to run a test or two of an interesting condition (small investment of time and effort) [21-25] Report bugs MAKE sure that EVERY Stakeholder understand definitions
- 17. Киев 2017 Risk-Based implementation: Risk Mitigation Risk based testing in reality: How to make it works on your project? RESULT = Extend of testing assigned 1. System response > 20 ms under full load (performance) 2. Patches and releases won’t install correctly (supportability) 3. Incorrect exchange rate for currency pair GBP:CNY (functional) 4. User failed to log in and authentication failed (functional) 5. User find interface too complex (usability) RPN 15 4 2 8 20 Extend of testing C E E B O TCs Suite-15 Suite-18 Suite-1 Suite-2 CR-5 US-101 TT-10 US-5 US-65 NA US Issues I-5, I-9 I-15 I-25 I-65 I-8 TRACING yes no yes yes yes Is Aut. Extend of testing 1-5 = [E] = Extensive 6-10 = [B] = Broad 11-15 = [C] = Cursory 15-20 = [O] = Opportunity 21-25 = [RP] = Report Bugs
- 18. Киев 2017 Risk-Based implementation: Risk Mitigation Risk based testing in reality: How to make it works on your project? Goal: allocate test effort [-] Mixed coverage [-] Catalogue Maintenance problems [-] Miss issues (RB, O) [+] Keep test FOCUS on WHAT is METTER [+] Limit test afford with RNP 1 test case covers ONLY 1 entity Automate Risk Catalog Establish Team Rules Allocate time for Experience based testing New project & Ongoing Project How to build Report with JIRA: Atlassian Support/JIRA Software Cloud/JIRA Knowledge Base Atlassian Marketplace: qTest for JIRA Test Management
- 19. Киев 2017 Risk-Based implementation: Risk Management Risk based testing in reality: How to make it works on your project? Goal: analyze, monitor and control [-] Catalogue Maintenance [-] Process Maintenance [+] Time Constraint problem Resolved (breadth-first and depth-first) [+] Bug-fix efforts focused on high-risk problems [+] Now we known the Scope Use Retro Build and Analyze Test Report based on Risks Communicate the process New project & Ongoing Project
- 20. Киев 2017 Risk-Based implementation: Metrics Risk based testing in reality: How to makes it works on your project? Goal: use metrics to improve the process • % of executed test cases per Risk item • % of failed/passed/unexecuted test cases per Risk item New project & Ongoing Project Test Execution metrics % defects identified in 1 Risk item % defect identified/fixed per one Risk item in one iteration % defect fount in the 1st – 2nd – 3rd Iterations – compere Defect metrics to establish Acceptance ‘Go’ ‘Not-Go’ Defect metrics
- 21. Киев 2017 POSSIBLE NEGATIVE RISK + = LIKELYHOOD IMPACT RPN x = HOW: Risk-Based testing process looks like? Risk Identification Risk Assessment Risk Mitigation Extend of testing Extensive Broad Cursory or or Opportunity or = Risk Management Execute Assign New RisksFix Report + + +
- 22. Киев 2017 WHAT: values Risk Based approach brings? Risk based testing in reality: How to make it works on your project? team knows the focus and domain “find the scary stuff first” “pick the right tests out of the infinite cloud of possible tests” “release when risk of delay balances risk of dissatisfaction” “give up tests you worry about the least” All these BENEFITS allows: the test team to operate more efficiently and in a targeted fashion, especially in time- constrained and/or resource-constrained situations
- 23. Киев 2017Risk based testing in reality: How to make it works on your project? ira.zhylinska@gmail.com
Editor's Notes
- #4: Is Risk-Based testing approach? Values it brings to both Project and Client
- #5: Possibility (Вероятность возникновение) Степень негативности Вероятность: То что являет фактом – точно не риск это ограничении what is impossible is not a риск и то что достоверно это ограничение Ну например мне не нужно волноваться что я сейчас поднимусь и ударюсь головой о потолок – тк гравитация работает или невозмножно не является риском– ( например гравитация она работает) не нужно волноваться совсем
- #6: Possibility (Вероятность возникновение) Степень негативности Вероятность: То что являет фактом – точно не риск это ограничении what is impossible is not a риск и то что достоверно это ограничение Ну например мне не нужно волноваться что я сейчас поднимусь и ударюсь головой о потолок – тк гравитация работает или невозмножно не является риском– ( например гравитация она работает) не нужно волноваться совсем
- #8: In risk-based testing, the results of Risk analysis will form key inputs into the test planning process as well as the test estimation process.
- #9: In risk-based testing, the results of Risk analysis will form key inputs into the test planning process as well as the test estimation process.
- #10: [+] Business availability [+] No defined Processes значит что мы можем его настроить так как удобно нам [+] Time – ВРЕМЯ вначале проекта у тестировщика есть ВРЕМЯ на то чтобы заниматься риск бейзд тестированием Identify + Communicate Present риск бейзд стратегию Combine – комбинируейте подходы, покуда процесс еще не установлен его можно рисовать так как прваильно и необходимо Learn - важно знание домена!!! Если домен сложный – тогда вовлечь команду – раздробить скоуп на куски и презентовать друг другу кусками Google – вся аналитика статистика все что может помочь при определении того как систему пользуют и что хотят от нее Кто юзера например? Основные фейлы предыдущих систем, Brainstorm Ask RIGHT questions in RIGHT place Внедриться в тылы БА для того чтобы учавствовать в груминге, если нет – получить доступ к сторям как можно раньше + сделать Жиру таким образом чтобы к сторям линковались десижены их договорится что все вопросы в десиженах БА отвечает + пока идет выяснение требований вики страничку где все будет документироваться именно под и для куа и риск бейзд
- #11: Find the RIGHT Guy человека/людей у которыъ есть доменные знания Привести пример Репортинговой апп (без доков и срочным релизом) Инвестигируйте проект (прежде чем менять нужно знать «где мы есть», «какие апроучи используем» «проблематики») Подготовьте решение для презентации и подготовьте план имплементации (пример Инвестмент проекта): стадия обсервации, подготовки, презентации и план итеративного внедрения Риск бейзд) и важно в отличии от например проекта нового мы презентуем решение уже решение, которое решает проблемы (ботелнеки)
- #12: Collect data from the past Trace past data with identified Risks Проанализировать предыдущие кейсы + автомейшен и намапить все на текущий чек лист
- #13: Collect data from the past Trace past data with identified Risks Проанализировать предыдущие кейсы + автомейшен и намапить все на текущий чек лист
- #14: Collect data from the past Trace past data with identified Risks Проанализировать предыдущие кейсы + автомейшен и намапить все на текущий чек лист
- #15: Особое внимание на 2е вещи
- #16: Analyse Past Expirience -> form RNP based on past data Два момента 1- хорошо включать в риск ассесмент 2- но лучше оставить для корректировки чтобы не сбивать с толку команду
- #17: extensive testing effort takes broad testing effort takes ~ 25-35 % from total test design implementation end execution Cursory takes ~ 5-15 % from total test design implementation end execution takes ~ 5 % from total test design implementation end execution
- #19: [-] Mixed User Story coverage (1 Test Case covers few functional entity) What if 1 step failed? 1 test case covers ONLY 1 entity Automate Risk Catalog as much “automated” as possible Establish Team Rules to support Correct Reporting Allocate time for Experience based testing
- #20: [-] Mixed User Story coverage (1 Test Case covers few functional entity) What if 1 step failed? 1 test case covers ONLY 1 entity Automate Risk Catalog as much “automated” as possible Establish Team Rules to support Correct Reporting Allocate time for Experience based testing
- #21: [-] Inappropriate Maintenance = забыли промаркировать = потеряшки Communicate the process всегда напоминайте и проверяйте
- #23: Is Risk-Based testing approach? Values it brings to both Project and Client
- #24: Possibility – what is impossible is not a ris То что достоверно это ограничение или невозмножно не является риском– ( например гравитация она работает) не нужно волноваться совсем