Model based ttcn-3 testing of a mobile operator charging
0 likes292 views
This document summarizes a project using model-based testing (MBT) to test the charging subsystem (CS) of a mobile operator. Key points: - An MBT approach was used to generate automated test cases from a state machine model of the CS to test it more quickly and thoroughly than manual testing. - Over 12,000 test cases covering all transition triples were generated and executed in under 6 hours, finding 15 new errors. This significantly improved test coverage compared to manual testing. - The mobile operator was convinced of the benefits of MBT for reducing testing time and costs while improving quality compared to their previous approach. They plan to use MBT for testing in the future.
Model based ttcn-3 testing of a mobile operator charging
- 1. Model-based TTCN-3 testing of a mobile operator charging subsystem T3UC, Beijing July 2010 Dr. Andres Kull, Elvior Dr. Kullo Raiend, Elvior
- 2. System Under Test ► Customer: Estonian mobile operator EMT (www.emt.ee) ► SUT: post-paid data charging subsystem in EMT ► CS: provided by Ericsson (www.ericsson.com) ► CS is customized using EMT business rules Billing Subsystem Charging Provisioning accumulators GSM/WCDMA SUT: Charging Subsystem Network Data session
- 3. CS core functionality ► Subscribers are provisioned by billing subsystem ► Provisioning the charging rules to the subscribers ► According to the data sessions the subscriber account is credited by CS Billing Subsystem Billing Sybsystem Charging Provisioning Charging reports Provisioning accumulators GSM/WCDMA GSM/WCDMA Data usage SUT: Charging Subsystem Charging Subsystem Network Network Data session
- 4. CS testing bottlenecks ► Ericsson upgrades CS software from time to time ► Customer introduces new subscription packages or changes existing ones from time to time ► Manual testing is time-consuming and error-prone ► Time for testing the updates is usually very short
- 5. Customer’s hopes from MBT ► Shorten the regression testing time ► Increase the test coverage ► Avoiding risks caused by the human factor in executing huge amount of boring tests manually ► Reducing the amount of manual tests significantly
- 6. MBT workflow used CS model in Poseidon for UML CASE tool Generator TTCN3- test scripts Develops Defines coverage CS Requirements Analyses ► System requirements are modelled TTCN-3 test tool ► Tests are generated from models ► Resulting tests have predefined coverage SUT: CS
- 7. CS features under test ► Subscribers provisioning ► Changing subscription profiles ► Changing month and day ► Bonus data amount usage ► Free data amount usage ► Priced data usage within HPLMN limit ► Priced data usage in the case of unlimited HPLMN ► Data usage if HPLMN limit is exceeded
- 8. Main test scenario TTCN-3 test tool Network Billing subsystem SUT: Charging Subsystem (CS) TTCN-3 test tool Billing subsystem Network Charging Subsystem (CS) Provision a subcsriber Generate data session Ask charging accumulators for the subscriber Charging accumulators Test if the subscriber account is credited as described by charging rules
- 9. CS model ► State machine models the behaviour of CS ► Only few model states (forBonus, forFree, noCharge, Priced, LimitExceeded) ► 16 context variables ► Properties of subscriber profiles ► CS accumulators for different purposes ► 106 transitions ► Express CS and environment transactions ► Used for modelling the charging rules
- 10. Tests generation and execution Coverage Generation Test lengh Amount of Execution time (transitions) TTCN-3 (LOC) time All transitions 2 min 213 9 213 5 min All transition 57 min 1 672 22 765 24 min pairs All transition 18 h 49 min 12 807 89 191 5h triples
- 11. Detected errors ► Note: CS was quite well manually tested before the MBT started ► In total 15 new errors were detected ► Detected charging errors could spoil the reputation of the operator
- 12. Project process, time spent ► Beginning (94 h, 31%) ► Introduction to problem domain ► Setting up test environment ► Executing 1st generated test case ► Incremental development to cover the scope (142 h, 47%) ► Model updates Results Results Model Results Model Model ► Test generation analysis analysis uddate analysis uddate uddate ► Test execution ► Analysing results Test Test Test Test Test Test executionexecutiongeneration execution generationgeneration ► Refactoring and test generation for different subscription profiles (32 h, 11%) ► Creating of documentation (32 h, 11%)
- 13. Test automation specialists feedback ► Higher test coverage than manually scripted ► The tests building productivity increases significantly ► Long generated test cases detected otherwise hard to find errors ► Significant tests maintenance costs decrease is foreseen ► Requirements traceability and results analysis is the key issue that should be improved in MBT
- 14. Conclusions ► Manual to MBT reduces test quality decrease caused by human factor ► All transition test coverage tests allow quickly (5min) verify in regression test the past functionality ► MBT revealed significant amount of errors that were not disclosed in manual tests ► At the end of project the testing staff was convinced that CS is ready for taking into real use ► MBT is the technology that EMT will start to use
- 15. Thank you! More information: www.elvior.com Cut your sofware testing expenses Functional black-box tests automation