Open Source Test Workshop for CIOs, CTOs, Managers
3 likes1,063 views
The document highlights the benefits and options of open source test automation provided by PushToTest, focusing on improving testing efficiency and reducing costs for organizations transitioning from manual to automated testing. It discusses the various tools and methodologies available for conducting functional, load, and performance tests, alongside insights on addressing performance bottlenecks. Additionally, it outlines the value of consulting services, training, and ongoing support to enhance testing practices and achieve better results.
Open Source Test Workshop for CIOs, CTOs, Managers
- 1. Surface and Mitigate Performance Bottlenecks Open Source Test Workshop For CIOs, CTOs, Senior Managers Frank Cohen, +01 (408) 871-0122, fcohen@pushtotest.com June 8, 2011 Wednesday, June 8, 2011
- 2. Open Source Test Automation Slides on Twitter ‣ http://twitter.com/fcohen ‣ Please retweet ‣ Please follow me ‣ #ost 2 Wednesday, June 8, 2011
- 3. Open Source Test Automation About PushToTest ‣ Our Mission: We Test, For A More Reliable World ‣ Make It Easier To Move From Manual To Automated Testing ‣ Support Advanced Testing, including Agile, Load, Integration ‣ Open Source Testing (OST) For Everyone 3 Wednesday, June 8, 2011
- 4. XML, SOA, and Performance Web 2.0 Success Driving Business ‣ Every Organization Wants Modern IT Functions and User Interfaces ‣ Rich User Interfaces (RIA, using Ajax, Flash, Flex) ‣ SOA Composite Applications • Faster Time-To-Market • Increase Revenue • Cost Savings ‣ Service Enabled Applications • Deliver More Integration Faster • Always-On • Less Cost 4 Wednesday, June 8, 2011
- 5. XML, SOA, and Performance The Reality of Web 2.0 Development ‣ We Build Applications Differently Now ‣ Haze of New Protocols, Software Models, Data Formats ‣ Agile Often Means Shorter Build-Test-Deploy ‘Waterfall-style’ Cycles ‣ Many Still Trying “Test And Trash” Techniques ‣ Uptime Depends on Business Service Management (BSM) Testing to Surface Functional and Performance Issues Every 8 weeks 30% of the Application Changes 5 Wednesday, June 8, 2011 30% Tests Re-recorded
- 6. Open Source Test Automation Testing Costs ‣ Proprietary Test Tools Offer: Functional Test Load Test Functional Test Platform $19,000 Load Test Platform $20,000 Functional Tests $8,000 Simulated User Licenses $120,000 Protocols: AJAX, SOAP $12,000 Recoding The Test $35,000 Support $5,000 Re-quality The Test $12,000 $40,000 $187,000 ‣ QA Labs for 10,000 Users = $35,000 Per Year ‣ DoesCause Analysis Consultants Root Not Include: Cost of Test Employees, ‣ Does Not Guarantee Success 6 Wednesday, June 8, 2011
- 7. Open Source Test Automation Proprietary Testing Falling Behind ‣ Adobe Releases Testing APIs for Flex/Flash in 2009 ‣ JQuery, GWT, YUI, Extjs Each Make 5 Releases in 2010 ‣ Appcelerator Titanium Makes 3 Releases in 2010 ‣ Microsoft Draws 8,000 to Mix for StarLight in 2010 ‣ StarEast, Better Software, PyPy, JavaOne, Devoxx, OSBC, LinuxWorld Feature Keynote Talks On Ajax ‣ “HP Refundedcustomer in theFor The Ajax industry. Handler”, a PushToTest Our Payment automotive Protocol 7 Wednesday, June 8, 2011
- 8. Open Source Test Automation Open Source Testing Emerges W tness Lots Of Options Selenium HtmlUnit soapUI DBUnit Sahi Watir FlexMonkey FIT Windmill Hudson TestMaker jMeter ‣ Signs of Maturity:Logger, Recovery Scenario Manager, Object Iteration Engine, Record/Playback, Data Generation/Importer, 8 Recognizer Wednesday, June 8, 2011
- 9. XML, SOA, and Performance Depend On A Service Partner ‣ Repurposing: One Test Script For Functional Tests, Load and Performance Tests, and Business Service Monitors ‣ Components: Tests Run Even After Application Changes ‣ Deployment: Run Tests In Your Lab, Ours, or Both! ‣ Services: Add Expertise, Training, Tools To Your Organization ‣ Enthusiasm:Openparticipate and contribute to We Source Test Projects 9 Wednesday, June 8, 2011
- 10. Open Source Test Automation Manual Testing Process Results 4 1 Recommend Changes Explore Manual Testing Tes 3 t 2 Plan 10 Run The Test Steps Create Test Plans Wednesday, June 8, 2011
- 11. Open Source Test Automation Side Effects ‣ Exhaustion ‣ Fatigue ‣ Blurry Vision ‣ Night Sweats ‣ Insanity ‣ Hair Loss ‣ Acne ‣ Divorce 11 Wednesday, June 8, 2011
- 12. Open Source Test Automation OST Architecture Application Under Test PTTMonitor Web, RIA, BPM, SOA CPU, Net, Memory Functional Test Injector Test Authoring Test Bundles ScriptRunners Data Production Libraries SoapUI (Pro) MySQL, HtmlUnit Sahi, Selenium, SoapUI, Java csv files, rdbms queries Sahi (Pro) JUnit Class Framework Load Test Eclipse Java IDE Fitness Test Operation Console Languages: Java, Python Wiki Collaboration Production Designer Grid and Cloud Repository Results Analysis Record/Playback Test Deployment MySQL RDBMS Business Intelligence Tool Monitor Continuous Test Repository Bug Tracker Monitoring Integration Management 12 Wednesday, June 8, 2011
- 13. Open Source Test Automation An OST Cloud Thousands of virtual users Stores results of every TestNode perform real-life scenarios on Results DB transaction, step, and the Application Under Test monitored resource PTTMonitor Application Under Test Load Injectors 400+ charts identify the root Analysis cause to performance bottlenecks, functional issues Central location from which a TestMaker tester, developer, IT manager, CI Console service runs and monitors tests 13 Wednesday, June 8, 2011
- 14. Open Source Test Automation An OST Cloud Thousands of virtual users Stores results of every TestNode perform real-life scenarios on Results DB transaction, step, and the Application Under Test monitored resource PTTMonitor Application Under Test Load Injectors 400+ charts identify the root Analysis cause to performance bottlenecks, functional issues Central location from which a TestMaker tester, developer, IT manager, CI Console service runs and monitors tests 14 Wednesday, June 8, 2011
- 15. Open Source Test Automation Browser Compatibility Tests Application Under Test TestNode Selenium RC SeleniumHtmlUnit Selenese Test TestMaker Console 15 Wednesday, June 8, 2011
- 16. Open Source Test Automation ScriptRunners ScriptRunner Type Name (langtype) Designer Script IE, Htmlunit designerscript Selenium HtmlUnit Browser selenium Selenium RC IE, Firefox, Chrome seleniumrc Sahi IE, HtmlUnit sahi HTTP Archive Page Optimization HAR Selenium RC IE, Firefox, Chrome seleniumrc SoapUI SOAP and REST soapui TestGen4Web Web 1.0 Tests testgen4web Mozmill Mozilla XUL Tests mozmill Unit Tests jUnit TestCase Java, Jython, JRuby, PHP, Perl, Groovy DotNet .NET Tests dotnet Visual Basic VB tests vb 16 Command Line Desktop and utils commandline Wednesday, June 8, 2011
- 17. Open Source Test Automation Component Approach TestScenario Component Application Log-In Load Test 100 Users 500 Users 1000 Users setUp Data runTest Test Use Case Database Production tearDown Development RDBMS DPL Log-In Database CSV DPL Product Search Component QA Regression Order Product Custom DPL Database Confirm Order Component Load Test Log-Out Database Production 17 Wednesday, June 8, 2011
- 18. Open Source Test Automation Lots of Possibilities TestMaker Test Sent Seattle, Datacenter TestNode Generating Load Dallas, Texas Datacenter You’re Such A Geek! 18 Wednesday, June 8, 2011
- 19. Open Source Test Automation TestMaker Demonstration ‣ Fun And Easy Test Script Creation ‣ Selenium and TestMaker ‣ Repurposing A Single Test ‣ Functional Test, Load and Performance Test, Production Monitor ‣ Results Analysis ‣ Chart Engine 19 Wednesday, June 8, 2011
- 20. Open Source Test Automation Root Cause Analysis ‣ PTTMonitor - Agent Based Monitor Gateway ‣ Glassbox - Performance Monitoring for Java Applications ‣ dynaTrace - Analysis In Application and DB Tiers 20 Wednesday, June 8, 2011
- 21. Open Source Test Automation PTTMonitor ‣ Correlate Test Operation To Resource Usage ‣ Deploy as Web Application (WAR) and Service ‣ Observes CPU, Network, Memory Utilization As Test Runs ‣ Gateway to Glassbox, dynaTrace, Windows PerfMon 21 Wednesday, June 8, 2011
- 22. Open Source Test Automation Results Analysis Engine 22 And 350 Additional Charts Wednesday, June 8, 2011
- 23. Open Source Test Automation Glassbox ‣ Correlate Test Operation To Application Tier Issues ‣ Memory Leaks ‣ Thread Deadlocks ‣ Slow DB Connections ‣ No Instrumentation of Applications ‣ Live Performance Alert View ‣ Run In Production 23 Wednesday, June 8, 2011
- 24. Open Source Test Automation Glassbox Troubleshooter 24 Wednesday, June 8, 2011
- 25. Open Source Test Automation dynaTrace PurePaths ‣ PurePath from TestMaker Virtual Users To Application & DB Tier ‣ Answers: What Are The Transactions and Why Are They Slow? dynaTrace monitors applications for functional issues dynaTrace Database Agent and performance bottlenecks. dynaTrace Application Server Agent PurePath Token TestNode TestNode TestNode test test test PTTMonitor Test RunTime 25 TestMaker Console Wednesday, June 8, 2011
- 26. Open Source Test Automation Pinpoint The Problem ‣ Down to the Line of Source Code 26 Wednesday, June 8, 2011
- 27. Open Source Test Automation .NET Monitoring ‣ Correlate PerfMon Statistics with Test Operation ‣ PTTMonitor Gateways Data from PerfMon or MS SQL Server .NET Application Tier PerfMon MS SQL Server SOAP HTTP REST TestNode TestNode TestNode PTTMonitor test test test Test RunTime TestMaker Console 27 Wednesday, June 8, 2011
- 28. Open Source Test Automation Next let’s talk about PushToTest Options ‣ OnDemand ‣ QuickStart ‣ POC Option 28 Wednesday, June 8, 2011
- 29. Open Source Test Automation Immediate Benefits ‣ Product Solutions ‣ Better, Lower-Cost, More Efficient Test Tool ‣ Business Service Monitoring at the Application Tier (SLA Compliance) ‣ Methodology Solutions ‣ Improved Test Methodology for my Organization and Team ‣ Services ‣ Turnkey Load Testing ‣ Integrate TestMaker Into My Build Environment ‣ Add A Protocol, Chart, Task to TestMaker To Meet My Needs 29 Wednesday, June 8, 2011
- 30. Open Source Test Automation Dual License ‣ Functional Testing with TestMaker Object Designer ‣ TestMaker Enterprise expands on the success of ‣ TestMaker Community Repurposing Selenium, soapUI Sahi and unit tests into data-driven functional tests, load and performance ‣ Load Testing to Millions of Virtual Users ‣ tests, and business service monitors Cloud and Grid Testing ‣ Load Testing To 50 Virtual Users ‣ Data-driven tests using Data Production Libraries for ‣ Data-driven tests using CSV files. CSV files, RDBMS, Data Generating Objects ‣ PTTMonitor Correlates CPU, Network, Memory ‣ Root cause analysis and mitigation using Glassbox ‣ Utilization As Tests Operate. Mock services for agile service development ‣ 25 Charts and Reports ‣ Flex AMF Protocol Handler for Flex/Flash testing ‣ Community Support and access to all documentation ‣ 1000+ Charts and Reports ‣ Business service monitors, including email notification ‣ Support, QuickStart, Training, Consulting 30 Wednesday, June 8, 2011
- 31. Open Source Test Automation Pricing ‣ Annual Subscription Model Based On: ‣ Seats for TestMaker Console Installation ‣ Seats for SOAP and REST Test Authoring ‣ Virtual User Levels for Load Testing ‣ Count of Business Service Monitor ‣ Technical support: 1messagesday or faster response to emails, phone calls, instant business 31 Wednesday, June 8, 2011
- 32. Open Source Test Automation ‣ As-Needed, On-Demand Basis ‣ Test In Your Environment, Or Our Cloud Test Environment, Or Both ‣ Ideal For Consulting Organizations Where Test Needs Are Unknown ‣ AMI Distribution ‣ Pay Only For What You Use ‣ Annual Cap On Total Costs 32 Wednesday, June 8, 2011
- 33. Open Source Test Automation The QuickStart Option ‣ Your First Test, Built and Operated by PushToTest Experts ‣ Ideal For Organizations With Little or No Load Testing Experience ‣ PushToTest Experts Build Your First Test ‣ PushToTest Experts Operate The Test, Provide Root Cause Analysis and Mitigation ‣ PushToTest Trains Your Team To Build Their Own Tests 33 ‣ $9,750 USD Wednesday, June 8, 2011
- 34. Open Source Test Automation The POC Option ‣ PushToTest Experts Help Your TestMaker Evaluation ‣ Ensures TestMaker Works In Your Application/System ‣ PushToTest Issues Price Proposal ‣ Applies Engineers To Build and Operate Your Test ‣ PushToTest Trains Your Team To Use TestMaker ‣ Success of POC Confirms Intention To Purchase 34 ‣ Cost: $0 Wednesday, June 8, 2011
- 35. Open Source Test Automation PushToTest Global Services ‣ Put Our Experts To Work For Your Projects ‣ Methodology, Solutions, Support, Enthusiasm ‣ Offices: USA (Texas and California,) Bangalore, London ‣ The Near Shore Option ‣ 150 Test Experts ‣ Costa Rica: CST, English, Inexpensive, On Site in 3-5 Hours ‣ http://www.pushtotest.com/services “PushToTest Uniquely Built Load Tests For Our Customers, Including Pepsi, And Worked With Our Engineers To Surface and Solve Performance Bottlenecks.” Alan Rodriguez, Director of Testing, 35 Tribal DDB Dallas Wednesday, June 8, 2011
- 36. Open Source Test Automation Slides on Twitter ‣ http://twitter.com/fcohen ‣ Please retweet ‣ Please follow me ‣ #ost 36 Wednesday, June 8, 2011
- 37. Open Source Test Automation Where To Go From Here ‣ Please Complete The Evaluation Form ‣ And We Will Send You The Slides ‣ Watch A Screencast and Tutorial ‣ http://www.pushtotest.com/screencastcentral ‣ Attend An Open Source Test Workshop ‣ http://workshop.pushtotest.com ‣ Ask For A Proposal on Licenses, Training, Support, Consulting ‣ Call +01 408 871 0122, sales@pushtotest.com 37 Wednesday, June 8, 2011