Testing a SaaS Platform
1 like4,036 views
Joel Montvelisky is a co-founder and product architect at PractiTest, a QA instructor and consultant. His presentation covers topics related to testing Software as a Service applications, including things that don't need to be tested for SaaS, non-trivial aspects to consider like security and live updates, and leveraging real user data to improve testing. He emphasizes that while SaaS changes the delivery model, customers still expect functionality, stability and performance, and outlines approaches to incorporate feedback from production systems into the testing process.
Testing a SaaS Platform
- 2. About me - Joel Montvelisky PractiTest – Co Founder & Prod. Architect QA Instructor & Consultant Mercury Interactive - QA Manager (retired ) TD, QC, WR, QTP, etc ITCB (IL) Advisory Board QABlog.practitest.com © 2010 -
- 3. Today’s Agenda SaaS - Software as a Service Things we Can Stop Testing Non-trivial Stuff to Think About & TEST Leveraging Live Feedback into the Testing Lifecycle Open Season for Questions… © 2010 -
- 4. What is SaaS? © 2010 -
- 5. Software as a Service Hosted Platforms – Total Control – No Maintenance No Client Installations Customers buys each month – keep them happy Users don’t care it is SaaS, it needs to be better than the alternative © 2010 -
- 6. SaaS QA Management End-to-End QA Management (Reqs – Tests – Bugs) Enterprise Level SaaS Solution - Methodology - Customizations & Flexibility Global platform - Communication - Languages - Available 24x7 Worldwide Intuitive & Simple Agile Dev, RoR, on Amazon EC2 © 2010 -
- 7. What does SaaS mean to the Testing Process? © 2010 -
- 8. Things we can STOP Testing ✘ Client or Server installations ✘ Multiplatform backend support ✘ Multiple version upgrades “Savings” of approx. 1/3 ✘ Backwards compatibility of the Traditional Enterprise Application Testing Cycle © 2010 -
- 9. The regular testing scenarios Functionality - Manual scripts - Exploratory sessions Agile Sprints & Updates - Checklists of 3 – 4 weeks - End-2-End Scenarios - Selenium Each sprint’s cycle is - Cucumber & RSpec based on its content Load & Stress Multi-platform & Multi-browser support I18N English, Swedish, German, Chinese Simplified, Hebrew © 2010 -
- 10. Non-trivial stuff to think about & TEST 1. Remote accessibility & usage Staging Servers in USA - Amazon EC2 Development & Testing teams in Israel (& WW collaborators) Globally distributed monitoring services (www.pingdom.com) © 2010 -
- 11. Non-trivial stuff to think about & TEST 2. Security Application-related: Infrastructure-based: Cross-site scripting Secured communication SQL injections Backups & storage policies HTTP header injections Controlled Access to the site & servers etc… © 2010 -
- 12. Non-trivial stuff to think about & TEST 3. Live updates & deployments (3 to 4 weeks) No concrete separation between Dev & IT Minimal service disruption Dressed rehearsals (including rollbacks!) Automated sanity & manual verification Patches & hot-fixes (extraordinary occasions ) Prove of Concepts & Beta Releases QA & Testing as a facilitator for Balance: Stable & Professional, yet Flexible Environment © 2010 -
- 13. Non-trivial stuff to think about & TEST 4. Disaster recovery procedures Two Main scenarios: (a) System down to be brought up quickly Configure machines Install & deploy software Restore data Define a policy & schedule regular drills to ensure you can achieve it! (b) Rollback to last known stable data (with or without data restore) © 2010 -
- 14. Leveraging Real Data into the Testing Process © 2010 -
- 15. Levering live feedback into the Testing Process (1) Internally developed monitoring tools - Check usage patterns No need to - Validate need & usability assumptions continue guessing!!! - Analyze patterns and create realistic & prioritized testing scenarios © 2010 -
- 16. Levering live feedback into the Testing Process (2) Site Monitoring systems LiveAdmin (www.liveadmin.net) Real-time usage to schedule updates Allow users to contact the company with questions or issues Pingdom (www.pingdom.com) Up to date response times for real projects and data. Proactive alerts when things start slowing down © 2010 -
- 17. Levering live feedback into the Testing Process (3) Proactive mail notifications on issues (www.hoptoad.com) Real-time notifications on issues on the system. Information about the issue including system parameters that allow you to do an initial trouble-shoot of the issue © 2010 -
- 18. Levering live feedback into the Testing Process (4) Ask users for Real Projects No better way to test than using real data Most users will trust you not to abuse their information in exchange for assurance of working software Since the data sits in your system it is easier to copy projects to your testing environment Need to be careful with who has access to the data; making sure your testing environment is as secured as your production environment © 2010 -
- 19. Wrapping up… © 2010 -
- 20. Wrapping up SaaS is only the delivery model, customers will not compromise on Functionality, Stability or Response Time Since the system is hosted you can save all tests related to backend installation and support You need to pay more attention to stuff like accessibility, security, deployments & DRPs The lines between the R&D and IT organizations in SaaS are blurry SaaS application allows better access to real data to be leveraged to improve the effectiveness of our testing operations. © 2010 -
- 21. Open Season for questions! © 2010 -
- 22. Special thanks to Anne-Marie Charrett joel@practitest.com http://qablog.practitest.com © 2010 -