The Journey of Test Automation
- 1. The Journey of Test Automation The Past Software testingisanecessaryprocessandeveryone seemstoagree onthat.Where people differ is the timing and depth of the initiative. The industry has long moved from a stage when testing was consideredapost-development,maintenance activity. We gradually interleaved the development and testing process. Both the development and the testing process tend to merge, bringing both newbenefitsandnewchallenges. The obviousbenefitisfastfeedback. We needtocheckan error as quick as possible. The not-so-obvious challenge is maintenance and overhead. This will come to haunt us later. Long since the early days, the industry adopted Continuous Integration. Whenever a developer is readyto pushsome code,he firsthas to pass itthroughthe UnitTests. If all isgreenthenonlycan he actually commit the code. Even per-commit jobs are gaining popularity. People are delegating the responsibilityof code validationtothe buildserveritself. The idea is to integrate the Code – Build – Test– Deploy processina single pipelinedworkflow. Everystage inthe pipeline providesimmediate feedback to the preceding stage. On the developers’ front automated Unit Tests have established themselves as an invaluable checkpoint. What about the managers and the business users? Everyone wants things to be well tested. But Unit Tests and their reports are not much meaningful for them. The unit tests are also not reliable in integration testing. Components having passed their unit tests may still fail to integrate.Here comesthe needof functionaltesting. Organizations have dedicated departments of manual-testers. They validate the product at whole build level. Detailed reports are generated, modules’ health is monitored and bug tracking is integrated to this process. The Present Bulk of functional testing has always been manual. A big reason was lack of tools which can automate applications, detect and report the failures in doing so. But the last decade saw an exponential growththe automationsector.The authorityof pioneering products was challenged by emerging tools. A majority of them were open source, promoting accessibility to even small companies. Sowhenwe can automate the functional testing process, why not integrate it to the CI server? Why not unit test and functional test every code batch? Here comes the hurdle which we knew but overlooked at the initial stages – maintainability. Automationscripts are traditionally codes. They are codes which test and validate other code; and they being code, need to be designed, written, maintained and reviewed. But they have a big advantage – theycan execute justasany othercode.That’swhyJUnit and similarframeworks are so popular.Problemistheydon’tscale wellwith functional testing. For one thing, functional tests are very broad, covering many workflows, scenarios, test cases and unique steps. Such big a code is difficult to maintain and soon outgrows the product itself. Another thing is being code; they don’t trulylie in tester’s domain. They need seasoned developers. You can have programmers write the test scripts but then a load mismatch remains.
- 2. Welcome the eraof KeywordDrivenAutomatedFunctional Testing.Encapsulating the programming processintestscripts givesmanybenefits.Firstandforemost,we nolongerneedhardprogrammers for testing task. Any programming-literate can do the task. The Test scripts are more maintainable and language agnostic. Now we can actually have both unit tests and functional tests which can be triggeredoneachcode checkin. While unittestsvalidate code conformity, functional tests validate product’s behavioral consistency. And both work unattended. Developers can get feedback meaningful tothemfromtheirunittests.Businessuserscangetmodule healthstatusmeaningful to them from their functional tests, all in near real time. The Future There is no limit to innovation. Imagine the development of a web application. Let’s say the application needs to be supported on all major browsers, IE, FireFox, Chrome, Safari, and Opera. Acrossall Platforms:Windows/Linux/Mac Desktops/Laptops. Let us push the limits, ask for support on all major mobile devices: Android, iPhone, Blackberry, Windows; talk about orientations, resolutions. Nowletsusaskthe testengineerstoautomate the whole thing.Runall regressionflows on all Platforms/browsers on all check-ins. After all we want a strict quality control. Difficulttoacceptbut thissimply cannotbe done!Notwithoutanautomation framework which can do the heavylifting.Noneof toolsorframeworksinthe marketcan cope withthe requirements that we will be facing in coming years. What the industry needs is a test automation framework which: 1. Can integrate with the existing CI solutions. 2. Performfunctional testsasseamlesslyasUnitTests.Run a testeverynight,everycheckin or on every Run-Test request. 3. Scriptingprocessisbetterto be keyword driven than language based. We want one simple ClickOnButton; not a bunch of find, hooks, performs, again recheck hook and so on. 4. Provides high maintainability of Scripts. Should not need programmers to modify that validation. 5. Can gracefullyorchestrate multiple tools to perform activities which otherwise don’t come under ambit of any one single tool. If Selenium cannot click on an image, get it done with Sikuli. 6. Can launchmultiple configurationsof asame testinparallel. We want to take the same test on all five major browsers. 7. Gives a uniform interface over heterogeneous tools. Tools may vary in language support evenonplatformsupportbutwe needit.We needto testour applicationon both Linux and Windows. 8. Generates business oriented reports. We want reports in terms of Business flows not individualtestcases.We wanthealthcheckof modules. We want basic metrics. We want to visualize defects over last five builds. 9. Drill down to step level failures, so that both testers and developers can get workable information regarding the failure. 10. Integrate with major Defect management tools like Bugzilla, Jira etc. Should be capable of automatically logging confirmed bugs. 11. Integrate with major Application Lifecycle Management software like HP ALM.
- 3. Now that is a whooping wish list and no product in the market claims to do it all. But this wish is soongoingto be our requirement.Successandfailure of products will depend on timely validation on all the target environments. Evenif we somehow manage the sheervolumeof test scripts and all the plumbing, we are stuck at the test infrastructure. We need to test on five browsers, each on latestversionandthree backwardversions.We needtotestthose browserson Windows, Linux and Mac. Nowthat makesabouta hundredparallel executionsonahundredinfrastructure.How are you goingto manage that infrastructure?Whathappenswhenyouneedabunchof Mobile devices? Are you going to purchase it all? Rent it all? Of course not. The future isof cloudcomputing. We will rentbrowsersoncloud.We will rentdevices on cloud. We will rent the execution systems on cloud. Volumes test data will be stored on cloud, analyzed on cloud systems and reported from cloud. All we need on premises is just one CI tool and one test- automation-tool. The cloudwill provideall the rest.We will rentresourcesonpay-per-usebasic. Pay for the tests we run. No need to inventory the hardware resources, browsers, devices and all. And our steps to it That future isnot so distant. Youcan get the machines fromIaaSprovideslike AmazonorRackspace. SauceLabs can rent you browsers; Xamarin can run your tests on rented mobile devices. These solutionsare already available. But are these true solutions to your pains? No. you need an end to endautomationsolutionandthese isolatedinitiatives don’t help much. A service is required which can bring all these together. A lot of tools have targeted the challenges and most have succeeded too.But until nowit hadbeenmostly isolated,unorganized attempts. Notable efforts are from QTP – which along with its Open Test Architecture is the big brother of the current generation of Test Automationsolutionsandthe otherisOpKey –the nextgenerationof Test Automation Framework. OpKeyisdetermined to solve the problems which the future automation engineers and managers are about to face. It is designed for scalability and maintainability. The future is of multi platform deploymentandsothe challenge lies in multi-platform simultaneous quality control. OpKey is the only player which can tackle this challenge. Being scalable means anyone, even small scale developerscanstart witha basicOpKeysetup andthenscale upas the applicationgrows. A time will come when a use case will span Web, Database, Web-service and Mobile interactions. At that momentyouwill realizethatonlyOpKey can orchestrate that many tools, platforms and resources. OpKeyisnotjust anotherframework.OpKey is the smartest Test Automation Framework. It adapts to your needs. It scales to your needs. It works for web, desktop and even mobile. It can simultaneously run all your tests on all the target platforms. What is more, you do not have to manage the entire hardware infrastructure yourself. OpKey is introducing Test as a Service paradigm. Withthat inplace,all yourinventorywoeswillbe gone. You will trigger the tests and you will getthe results. That is all. OpKey is the beginning of no-frills-testing. In next few articles I will demonstrate how it meets all that it claims.