Automated Testing With Jasmine, PhantomJS and Jenkins

Editor's Notes

• #3: Who we are Testing JS code and front-end Listing the types of tests Looking at some test tools Writing and running tests Testing without the browser What is a headless browser Scripting with a headless browser Testing with a headless browser Continuous Integration Deploying code Integrating testing with builds Review build results
• #4: Who we are Testing JS code and front-end Listing the types of tests Looking at some test tools Writing and running tests Testing without the browser What is a headless browser Scripting with a headless browser Testing with a headless browser Continuous Integration Deploying code Integrating testing with builds Review build results
• #5: Who we are Testing JS code and front-end Listing the types of tests Looking at some test tools Writing and running tests Testing without the browser What is a headless browser Scripting with a headless browser Testing with a headless browser Continuous Integration Deploying code Integrating testing with builds Review build results
• #6: Who we are Testing JS code and front-end Listing the types of tests Looking at some test tools Writing and running tests Testing without the browser What is a headless browser Scripting with a headless browser Testing with a headless browser Continuous Integration Deploying code Integrating testing with builds Review build results
• #7: We're both developers from Work at Play. Work at Play is digital agency that provides clients with custom engagement solutions that decrease churn and increase long term customer value. This basically translates to either building apps and using social marketing, or both.
• #8: Just some of the Drupal sites we've worked on ABCSpark.ca BarbieCollector.com, Mattel.com, MonsterHigh.com, HotWheels.com (no image), Knowledge.ca NeedForSpeed.com
• #9: Some other CMS and frameworks: Yii, CodeIgniter Java Spring, Alfresco Django PhoneGap Not even including some other ones, like Kohana and TYPO3 Then there are the JS frameworks: BackboneJS KendoUI ExtJS jQuery (UI, Mobile)
• #10: Now, regardless of the backend, framework the result is lots and lots of HTML and JS (and CSS).
• #11: [pause for about 20 or so seconds] / "I'll let you read this comic first" You could test your pages via manual click tests, but that's ridiculous. You have to do these manual tests every time your code evolves with new features, or improvements, to ensure that your code is still working as it should. Which can become frustrating with the amount of time and effort an individual person has to put in in order to do so. So why not automate it and remove that pain? Hence, this session on AUTOMATED TESTING WITH JASMINE AND PHANTOMJS.
• #12: This is our company slogan. It is painted on the wall We work smart partly by automating tests... This gives us the opportunity to play hard with new and interesting technology
• #13: Saves time and money test to ensure quality but Manually repeating these tests is costly and time consuming. automated tests can be run over and over again at no additional cost and they are much faster than manual tests Improves accuracy tester will make mistakes during monotonous manual testing Automated tests perform the same steps precisely every time they are executed Increases test coverage automated testing can increase the depth and scope of tests to help improve software quality Lengthy tests that are often avoided during manual testing can be run unatten Does what manual testing cannot simulation load and stress tests Helps developers and testers developers can share test codes; tests can be ran by developers and not just testers Improves team moral with the the time saved, the team can work on more building and solving interesting problems instead of manually testing over and over again and continuous integration is the cornerstone of Agile development Agile practices rely on fast feedback, achieved by testing each incremental change Automation allows us to break free of waterfall's "coding phase / testing phase" paradigm, turning them into an integrated development cycle
• #15: What we have here is a lit of types of tests. Of course, there are other types of tests, such as stress tests, but they're out of scope. As you can see, unit tests are at the bottom of the list because they're lower level tests compared to GUI tests. The test pyramid is a concept developed by Mike Cohn , described in his book Succeedin g with Agile . It's a pyramid because you have much more low level tests at the base. And as you go up, there are less and less tests of that type. Also, the higher level tests can encompass one to many lower level tests.
• #16: On the front-end or client side, the way these tests are normally done are: GUI - automating the browser by scripting it or recording actions Unit test - by writing and running tests in JS
• #17: Now, here are some testing tools; there are many more obviously. In this list, the ones at the top like selenium and watir does testing by automating the browser. The ones below are mainly for unit testing your JS/HTML code. Some tools, like Windmill and Mocha, can do a both.
• #18: In this session, we're obviously be looking at Jasmine, but we'll also briefly look at Selenium, which sits on the other side of the spectrum.
• #19: Compare with selenium. More info on selenium. Highlight what selenium does better than jasmine and vice-versa.
• #20: Many other javascript testing frameworks only work from within a browser. Some don't support testing asynchronous code like event callbacks and some still have syntax that's hard for JS developers or IDEs to understand. Jasmine aims to run on any JavaScript-enabled platform, to not intrude on the application nor the IDE , and to have easy-to-read syntax. Jasmine was designed with principles in mind for what makes a good up a JS testing framework. With the goals for writing jasmine, [pause] one being that it should encourage good testing practices. [next slide]
• #21: More about what Jasmine is, and what are some of those principles and goals... Well first, Jasmine is a behaviour driven development framework for testing javascript code. [next slide]
• #22: - A Behaviour driven development framework is a specialized and continued version essentially of TDD - as it makes more specific choices than TDD does - it focuses on the desired behaviours of software, borrowing from agile software development - so tests are specified in terms of those "desired behaviors", using sentences starting with the word "should"
• #23: It is designed to not be tied to any browser, framework, platform, or host language. You don't need any other javascript frameworks or libraries in order to start working with jasmine. It only needs them if it is testing it. It also doesn't need the DOM, which means that we can do not need to rely on it to write unit tests and can just interact with only the javascript objects that we need and want to test. But though it doesn't require it.... [next slide]
• #24: ... it can still access the DOM if needed! It can work anywhere JavaScript can run, including browsers, servers, phones, etc. We still have the ability to test the DOM too, and can use any libraries or frameworks we choose to.
• #25: Jasmine is designed with the principle that a good JavaScript testing framework should also have clean, idiomatic and unsurprising syntax in order for it to be easy to read and that writing jasmine should be simple to get started with.
• #26: Looking back the the test pyramid, we know Jasmine can do unit tests. Not requiring the DOM means we can test our JS objects.
• #27: With Jasmine's async support, we can test backend services and APIs by making AJAX calls.
• #28: Having access to the DOM means being able to test the UI, e.g. check if certain Drupal blocks exist on a page, fill in a form and submit it, and so on.
• #29: So, lets dive into jasmine and how to write these tests: Jasmine has the usual set of features (for a js testing framework), such as the following, which we will go over in more detail later. It has: - test suites and specs - the "expect" test method - various matchers which are included but also the ability to create your own custom matchers - setup and teardown functions - spies - asynchronous support - Plus many more... For this session, we'll look at using these to test a basic calculator that we set up in Drupal.
• #30: Here you see the sample Drupal site and calculator.
• #31: On closer look, the calculator provides 4 mathematical operations and two input variables, that output the results of the chosen operation to the final field.
• #32: This JavaScript object implements our simple calculator.
• #33: Here we show how we utilize the calculator object. We have one or more form with the class 'calc' and we simply call 'set' and 'calculate' functions of the calculator object when certain events are triggered. What we'll do initially is to unit test the calculator object (shown in the previous slide).
• #34: [after setting up the HTML runner file that actually runs the Jasmine tests, which you can take a look at in the jasmine documentation or sample files] (First thing, ) We can create the test suite to hold all our tests, or "specs", for the calculator. It begins with a call to jasmine's "describe" function. You pass it two parameters, a string and a function. The first parameter, the string, is the name or title of the suite. The function is container for the block of code that implements the tests of the suite. This is where you would start to put in your specs.
• #35: A spec is a function very similar to a suite, it also has two parameters with a string and a function, Though it is defined by calling the jasmine's function "it". The string is the title of the spec and what exactly it is testing. As we mentioned before, since it is a behaviour driven testing framework, it is likely a statement that starts with "should" In this case "should be able to add two numbers together". The function is where the block of code to execute that spec is placed, and we begin to interact with our calculate object.
• #36: Expectations are written with the function expect. It takes a value, the actual, which we will implement matchers on to evaluate it.
• #37: Matchers implement a boolean comparison between that "actual" value and the expected value we compare to. Jasmine has a set of matchers included that we can begin to use, which we can see here. toBe: compares x and y and passes if they are the same toEqual: compares x and y and passes if they are equivalent toMatch: compares x to the string or regular expression and passes if they match toBeDefined: passes if x is not undefined toBeUndefined: passes if x is undefined toBeNull: passes if x is null toBeTruthy: passes if x evaluates to true toBeFalsy: passes if x evaluates to false toContain: passes if the array or string x contains y toBeLessThan: passes if x is less than y toBeGreaterThan: passes if x is greater than y toThrow: passes if the function fn throws an exception e when executed
• #38: But any matcher can evaluate to a negative assertion as well by chaining the call to expect with a not before calling that matcher. It compares the "actual" value to the expected and then passes if they are NOT equivalent (in this case).
• #39: In our calculator example, we use the toEqual matcher after we using the calculator object to calculate using two numbers (10 and 15) which were in turn ADDED together The result of that action we use as the "actual" value in the expect function then check that the expected value EQUALS in this case 25.
• #40: We can start adding more specs for other operations of the calculator within our test suite. Such as ones for subtracting and multiplying seen here. They both use the expect and toEqual matcher to see if the actual value returned from our calculator object is the value we expected. But alternatively, we could possibly use other matchers such as toBeLessThan for the subtraction spec, to test that the "actual" value is less than x. (TODO: Should this be another slide?)
• #41: The next feature is setup and teardown. These are functions are called before and after each test within a test suite. Here you see the basic syntax for the setup, which is function beforeEach. Then the teardown, which is function afterEach.
• #42: We can use the beforeEach to help set up variables or the state of the code we want before each of our tests. So now we don't have to repeat that code in every spec we write. For these tests, we use the setup to create a new instance of our 'calculator' object.
• #43: Another functionality to use is Spies. Spies act as test doubles and can sub any function and track calls to it with all the arguments. Spies have their own set of special matchers to be used to interact with them. Here we set a spy to make sure that the function 'add' is called when the function 'calculate' is executed with the "+" symbol. It should be noted that we can no longer do expects on the value returned for z, as the function "add" was subbed out with the spy. Which means that when add is called by calculate it doesn't call the calculators add function which returns the calculated value. So we no longer check if the value of z is correct in this particular tests. Hence why the spy is in a separate test than those expectation tests.
• #44: Now, we can move onto testing the calculator UI. This time we will interact with the DOM using jQuery for our tests. (We will need to remember to include jQuery in our runner HTML file to do this.) In this test: we get the form populate fields with values submit the form by triggering a change then check the resulting value of the answer field When the test pass, we know a few things: our form and input fields do exist the event handlings are good and the calculator object still works while implemented with the DOM
• #45: We can add the DOM tests to the rest of our calculator test suite. To do this, we can nest describes within our main test suite: we can group these DOM specific tests together we can add it's own set of beforeEach/afterEach functions for this set of tests
• #46: Here we use the beforeEach to set the form variable that we use it our DOM tests. But we also start using the afterEach as well. We use it to reset the form and it's input fields after we finish each test. So the next test we write starts fresh and doesn't have values that could mess up its results.
• #47: Jasmine also has support for running specs that require asynchronous operations. Specs can be written with a set of blocks with calls to the function "runs" and "waitsFor". For our calculator example, it does not have any operations that are asynchronous, so here is another example using a timeout to demonstrate this functionality. The first runs holds the asynchronous call, in this case the timeout function.
• #48: The waitsFor block has the latch function, which polls until it returns true. Which we set with the timeout in the runs section above. The waitsFor block also takes an optional failure message and a timeout. If the timeout for the waitsFor expires, the spec fails with the error message.
• #49: But once the asynchronous conditions are met based on the return in the latch function of waitsFor, another "runs" block defined the final tests. Which is usually the expectations based on the state after the original async call returns. For example, asynchronous operations is very useful when testing things like: - code that requires loading, for instance: downloads, video or music players, and many others - or code with animations that you may have to wait for it to finish before testing the state after that animation - or when submitting forms - or even testing ajax or other api calls (via javascript)
• #50: The javascript file with the suite, is then included via a <script> tag within the HTML runner file. The runner file is used to get Jasmine to execute the suite or suites and all its specs, then use an instance of a reporter such as the HTMLReporter, which is included with jasmine, that calls to get the results of each spec and each suite and finally present those results to the user. The HTMLReporter displays all the suites with each individual sub-suite and spec that ran. It displays in green when the test passes, or red if it fails. Plus it gives a final count of the number of specs that ran, and how many were successful. In this case, we ran all the calculator specs within the suites, totally 6, and all passed!
• #51: The javascript file with the suite, is then included via a <script> tag within the HTML runner file. The runner file is used to get Jasmine to execute the suite or suites and all its specs, then use an instance of a reporter such as the HTMLReporter, which is included with jasmine, that calls to get the results of each spec and each suite and finally present those results to the user. The HTMLReporter displays all the suites with each individual sub-suite and spec that ran. It displays in green when the test passes, or red if it fails. Plus it gives a final count of the number of specs that ran, and how many were successful. In this case, we ran all the calculator specs within the suites, totally 6, and all passed!
• #52: We can also delegate filtering of the specs to the reporter. This allows for us to click on individual suites or specs and run to see that particular subset, and not the entire list of all tests. We can choose to rerun just the "should be able to add two numbers together" test when that is the only one we care about at the moment.
• #53: TODO: Image: Possible idea is to have the code for the test suites then a funnel to go into the headless browser? Now we have a suite of tests for our calculator code that we can run in the browser, we can check if our code works and continues to run as expected. But this still is partly a manual process. The test pages are in HTML and need a browser to run it, with a person still that is to initiate the process and run that page manually. To further automate this process, we want to start to be able to have the tests run but outside of a browser now, something that a non-human could execute. Such as in the command line with the use of a headless browser.
• #55: [Useful for]..... particularly for AJAX heavy web sites/apps
• #56: So, there is PhantomJS! It is a Headless browser, specially a headless webkit. Which is essentially LIKE Chrome, But - no window - no browser, - and an invisible viewport [Accordingly] It has a javascript API, and so you can feed scripts written with javascript or even coffeescript into the command-line based utility. It has support for various web standards such as: DOM handling, CSS selector, JSON, Canvas, SVG... Really, anything that webkit can do!
• #57: So, let's try out a basic example of how to execute JS based programs with PhantomJS. We start off by creating a script file, in this example called hello.js With javascript we can output a message from it using console.log. Then tell phantom to stop and exit To run the script that was just created, we call phantomjs in the command line and pass the path to the script file we just created. It runs our script, and as instructed outputs the message we specified.
• #58: PhantomJS uses Module API, which is modeled after CommonJS Modules that are available. This set here is available up through PhantomJS 1.6. ... [read slides]... [Webpage] (read above) is an object that encapsulates a web page (and its functionalities) [System] holds the set of function to access the system level functionality... which includes the os, platform, and env, along with access to the arguments passed into the script [FileSystem] (read above) with the ability to read and write [WebServer] on the other hand is experimental. It uses an embedded web server module called Mongoose, which phantomJS can script to start a web server with. This is intended for ease of communication between PhantomJS scripts and the outside world but is not recommended for use as a general production server.
• #59: So using those modules we can use phantomJS for things like: Page loading This example shows a script called loadpage.js In the script, we load and create one of phantomJS's webpage objects. We use the object to load the URL to drupalcon portlands website, using page.open for the first parameter. In the second parameter is the function which is called after the page is loaded, or finishes attempting to load. We explain this a bit further in the next slide. After the page loads though in this case, we render the website to a png image called drupalcon.png Then finally exit from phantomjs
• #60: But to further extend the page loading script, we want to pass in arguments and tell our script dynamically which url we want it to load in a page. In order to do so, we take the basics of the last example and modify it.
• #61: We start by also creating one of PhantomJS's system objects after the page.
• #62: Then we get the first argument, AFTER the path to the script that we are running, from the system.args. The arguments include the path to the script as the first argument passed to phantomJs. We use the URL argument passed to be loaded into the WebPage object
• #63: As mentioned before, the second parameter is a function that is called after a page is loaded, but is really called after the attempt to load has happened. So it will also be called if the page failed to load. It passes 1 argument to that function for the status of the page load. We can use it to see if the page load was successful or not, then do different actions depending on it. Here, we output to the console that the address was loaded if it was successful.
• #64: Now we can run this example, much like we did before. We call phantomjs and pass in the script path as the first argument, but then as the second argument we pass in the URL that we are going to load. For example, we pass in the URL for google. PhantomJS runs our script, using the google.com parameter to load that page and when it successfully loads, outputs to the console window the message that the address was loaded.
• #65: Using PhantomJs's webpage object, we can also conduct code evaluation on the page that was loaded. After a page was successfully loaded, we can use phantom to access and manipulate the code on that page within the page.evaluate function. From there we can return elements or information from the page loaded, for instance the document title, and set it to a variable within our phantomjs script. Then output that title to the console when we run it.
• #66: Page loading and code evaluation are used in conjunction to run the jasmine tests that we created, then parse the results from the page so that we can output them in a way that is useful for automation. Thankfully, phantomJS has a list of example scripts, one of which is to run tests with jasmine, which you can find on github under the examples folder. So we can take a look now at the code for run-jasmine.js
• #67: This example, loads the page to the jasmine runner file that runs the tests which we pass in as the first argument after this script path. Looking at the code we see the what we learned before with page loading....
• #68: the webpage object being created....
• #69: then opening that webpage with the variable to the jasmine tests. After page loads successfully, the script needs to wait for the tests to finish executing.....
• #70: It does this with its own "waitsFor" function which you pass the check for the condition of what we are waiting for as the first parameter...
• #71: ...it uses code evaluation with page.evaluate to do this....
• #72: ...and test when the pending class on the body is removed. Which is done by jasmine's HTML Reporter after all the tests done running.
• #73: Here we can take a bit closer look at this WaitsFor function. [next]
• #74: Not only does it pass the waiting condition, but you also pass it a "onready" function that is called when the waiting condition is met...
• #75: and an optional time in milliseconds for how long the script should wait for that condition to be met. By default it is set to 3 seconds.
• #76: When that time is reached, and the waiting condition is still not returned, it tells the script to stop, and exit with a failure.
• #77: Otherwise, it calls our onReady function that we passed in.
• #78: Here we see that onready function to be called.
• #79: it uses page.evaluate again to access the page the DOM of the finished runner HTML.
• #80: It parses the HTML to identify the number of successes
• #81: and failures. Which are reported to the console.
• #82: Depending on if there were any failures, it exits phantom with the exit status of 0 or 1. 1 being if any errors existed.
• #83: So in conclusion, lets take a quick look at some of the features that we went over for PhantomJS and what it is capable of. You can do a lot more with phantomJS outside of using it to run javascript tests with jasmine. phantomJS can do fast headless testing with other test frameworks too such as Mocha, Qunit, WebDriver, etc But using what we have shown today, it can be used for many different possibilities. You don't have to stop at automating tests! We discussed how it handle Page Automation by accessing and manipulating web pages with the DOM API. But it can also do this with the usual javascript libraries like jQuery. It can also be used for Screen capture, to capture web contents including CSS, SVG, and Canvas For example, there is already a Drupal module out there using PhantomJS, called PhantomJS Capture It has provides integration with PhantomJS and the screen capture abilities to allow you to take screenshots of homepages and use them as images on your site. Finally, it is also possible to use it for Network monitoring. Such as automating performance analysis, or track page loading. You can also use YSlow to integrate with PhantomJS (and even Jenkins) which would be useful for automating web performance.
• #84: Plus With version 1.7, phantomJS also adds the possibility of creating and using custom modules. Users can reference and load their own modules using the require function in phantomJS scripts, just as they would with the included Phantom ones Seen here we use the require function to load the webserver, but also our "MyAwesomeModule" in the same way then call functions from the loaded object [pause] But let's head back to finish automating our tests...
• #85: So to summarize, we can now see how to execute the Jasmine tests, and run them in the command line with PhantomJS. We can easily run our tests as part of a script! Since we use Jenkins for deployments, we can run the tests after commits as part of the [Next slide] Continuous Integration.
• #87: Leading open-source Continuous Integration server Built with Java and it's easy to install and configure Plug-in support with over 400 plug-ins RSS/E-mail/IM support for notifications
• #88: Here's the dashboard of jenkins Currently mousing over the project and we see a dropdown menu.
• #89: Project view. Build history on the bottom left.
• #90: We clicked "Build Now" We see a progress bar. We can also see a more detail view of the progress by clicking the build in the build history
• #91: Here's the details of the build as it happens. We can click on the console output to see what the server is doing
• #92: Here's the console output We can see that it did a git pull
• #93: Right now, the script we created in teh previous section, we had the results from the jasmine tests get spit out with PhantomJS into the console window. For Jenkins to properly understand those results we need to convert the output into something that jenkins can use. The way to so that is to structure the results of the tests into XML. There is a couple version already on github that people have contributed, but the one we have used it from the phantomJs - Jasmine XML Example Project. It is an example project layout that is destined to be used standalone or with Maven, but also works with Jenkins! From this repo we essentially use 2 scripts, a new reporter, and an altered phantomjs script.
• #94: First, is a plugin for jasmine for a new reporter. It saves the results of the tests which the phantomjs script will interact with.
• #95: To enable it, first include the script in the header, but then before executing jasmine, replace the HTMLReporter, with the Trivial Reporter from Jasmine and then this new PhantomJSReporter.
• #96: Next, we take a look at the phantom JS script at phantomjs_jasminexml_runner.js We could take this script as is which is really close but uses different waiting functions, or we can the take parts of it we need and replace sections in our current script.
• #97: First, we change the code in the evaluate function for the waitsFor to instead, check the variable in jasmine was set from the new phantom js reporter we use instead of checking the DOM.
• #98: When it finishes, the waitsfor function needs to call new code for our on ready function
• #99: we replace it so that we can get the suite results from the new phantom js reporter instead of parsing the HTML of the page.
• #100: We loop through these results, which returns a list with items that contain an: xml file name Which comes from the name of the suite that the results are for. For our script we will only have one, the one called calculator.
• #101: The result also include the xml body. It contains the formatted xml with all the specs for that suite: For each spec, it has the title, the passing status, and any possible error messages.
• #102: Our script can now takes that xml file name and create a new file using the PhantomJS file system. Then save content with the given xml body to the created file.
• #103: Finally, the script to exit changes a bit. It will get the passed status from the new reporter which will be used to return 0 or 1 again. so to summarize, with the changes that were made now when the script runs: it will run the jasmine tests, get the results of those tests with the new reporter and NOT the DOM, it will structure those results into xml and save the xml into files that jenkins can later use to parse to display the results.
• #104: Now we can setup Jenkins to use our new script during the build phase. On the confirguration page of Jenkins, we can select to execute a shell command after we build and push our code. Here we tell phantomjs to run our script, passing in the path to our script, referencing it from the jenkin project workspace. Then we will aslo pass in the URL to our jasmine test runner page, and finally a folder within the workspace for the script to save all the xml result files to.
• #105: Now when we build, and see the progress in the console output page, we can watch as the jasmine tests run which still show messages in the console for what tests are currently running, and whether or not they passed.
• #106: But for Jenkins to start using the results to display outside of the console ouput, we need to tell it to parse the XML files. We go back to the configuration page, and from within the post build actions section, we can add "Publish xUnit test result report" action, and select to use a JUnit Pattern. The resulting input field is where we enter that folder to all our xml files that we passed to our phantom script for the shell command configuration. We can leave the rest of the checkboxes as the defaults.
• #107: But now we can also choose to configure how many jasmine failures it should take to mark the build as having failed. In most cases we can set the threshold to 1 for all fields. So that if one or more failures happen during a build, that build is marked as broken. We can also use this to trigger events such as notifications if we needed.
• #108: But for now, we can see Jenkins parsing our results after the tests run in the console window.
• #109: Then when it is finished we can see the Test Result section on the main build page, with the count of failures in brackets beside it. (Here we see that there we no failures that happened for our build)
• #110: Then when it is finished we can see the Test Result section on the main build page, with the count of failures in brackets beside it. (Here we see that there we no failures that happened for our build)
• #111: If we click on the Test Result link, we can see more information about the tests that ran. The bar at the top shows the precentage of successful tests to failed tests. Blue for the percentage of successful ones, and red for the failed. If there were any failures, for each of them they would show in the table beneith All Tests. You could click on any of the failed tests, and see a page that displayed the error message from jasmine for why it failed. But in this case, all our tests were successful so we don't see anything here.
• #112: If we click on the Test Result link, we can see more information about the tests that ran. The bar at the top shows the precentage of successful tests to failed tests. Blue for the percentage of successful ones, and red for the failed. If there were any failures, for each of them they would show in the table beneith All Tests. You could click on any of the failed tests, and see a page that displayed the error message from jasmine for why it failed. But in this case, all our tests were successful so we don't see anything here.
• #117: .... END