Bart Knaack - The Truth About Model-Based Quality Improvements
Download as PPT, PDF0 likes740 views
The document discusses model-based quality improvement in testing, outlining various methods including online and offline model-based testing. It highlights the advantages and disadvantages of each approach, such as quick retesting and the necessity for model adapters. The conclusion emphasizes the potential of modeling in quality improvement, the need for new skills, and the lack of standards in the field.
Bart Knaack - The Truth About Model-Based Quality Improvements
- 1. © Logica 2008. All rights reserved The Truth about Model Based Improvements Bart Knaack
- 2. Agenda • Introduction • The future of testing • What is a model? • Different ways of Model Based Quality Improvement • Conclusions • What next
- 3. Bart Knaack • Bart Knaack, Senior Test Advisor, Logica, The Netherlands • 15 years experience in IT, of which 12 in testing • Father of 2 kids (age 6 and 8) Bart.knaack@logica.com
- 4. The future of testing • Crystal ball called Academia • The past as the mirror of the future • Trends and threats But: • Testing is not quality improvement • Testing is not LEAN
- 5. What is a model? A model is an abstraction or conceptual object used in the creation of a predictive formula (wikipedia) A model in science is a physical, mathematical, or logical representation of a system of entities, phenomena, or processes. Abstract representation of the reality focussing on a limited set of aspects Programs are models too!!
- 6. Model for this presentation Behavioral models: Models that somehow define the (intended) behavior of a system •UML models •Finite State Machines •Algebraic models •Graphical model •Textual model
- 7. Model Based Quality Improvement No. 725 March 2015 Testing a Virtual Juggling Program
- 8. Different ways of Model Based Quality Improvement? • Model Based testing –Online –Offline • Model Metrics and Code Conformance • Model checking the models • Applying Model checking techniques.
- 9. Model Based testing (Online) • Definition: Directly coupling the test generation model to the SUT and directly executing the tests. • Project: IRIS DOMAIN: Insurances SUT: Actuarial calculation engine. Torx modelling language for modelling the formulae. Result: 5 Additional errors were found of which 3 were significant. Development and test-time for upcoming changes and fixes was reduced by 50%.
- 10. Model Based testing (Online) • Advantages: –Quick (re)testing of systems • Disadvantages: –Model SUT adaptor needs to be set up. –Model errors need to be taken into account –Not all systems and tests are suitable for this approach.
- 11. Model Based testing (Offline) • Definition: Tests are generated by a Testgenerator and executed manually on the SUT. • Project: CCBS Domain: Telecom. SUT: 5ESS Telephony switch, new feature Result: 10 features Modeled.1200 Tests designed of which 75% specific on feature interactions. 20% tests failed due to modelling errors.
- 12. Model Based testing (Offline) •Advantages: –Quick adaption of testcases –Models can be used for rapid prototyping. –Quick reponse on Changes •Disadvantages: –Most methods do not support the selection of testcases. –Testexecution still manual
- 13. Model Metrics and code conformance • Definition: Using Metrics to determine when models are good enough. • Project:Siu Wai Tang Domain: Banking and Finance SUT: Electronic Payments System. Result: Improved Insight in elevated problem areas; redefinition of 3 models based on this insight.
- 14. Model Metrics •Advantages: –Early stage warnings on model quality. –Multidimensional view on models. •Disadvantages: –No hard figures to adhere to. –Statistical co-relation instead of direct cause-and-effect.
- 15. Model checking the models • Definition: Requirements checking by checking the models that form the requirements. • Project: Bos project Domain: Industry SUT: ‘Waterkering’ (lock) protocols. Result: Error free software (after 20 years of operation).
- 16. What is Model Checking? •Statespace search •Validation of a negative scenario. •Structural traversal.
- 17. Model checking the models •Advantages: –Finding CONCEPTUAL errors at an early stage. –Finding DIFFERENT types of errors •Disadvantages: –Model needs to be implemented in a formal manner –Model checker needs to be available for the used paradigm.
- 18. Applying Model checking techniques. • Definition: CODE is used model. Model checking techniques used for faultfinding • Project: FeaVer Domain: Telecom SUT: transaction based system (V5 interface) 5000 lines of code 1000 installments over 10 years. Result: 35 bugs found (25 race conditions)
- 19. Applying Model checking techniques. •Advantages: –Quick (re)test functionality –Model errors are REAL errors –Errors are found that are next to impossible to find using ‘traditional’ methods. •Disadvantages: –Not all languages are supported by modelcheckers. –Stubs and drivers need to be implemented to FEED the system.
- 20. Conclusions • Modelling by itself can be quality improving. • A lot can be gained from modelling and model improvement initiatives. • Model Based Testing can help in t ackling the problems in complex systems. BUT: • New skills • Academic image • No standards
- 21. What next ? • Integrated Toolsupport • Simplify • Educate • Not included: Model Based development.