A Top Down Approach to End-to-End Testing

Editor's Notes

• #5: -Cigna  Increased release frequency by 2x as a result of improved test automation -Bonotel  Achieved 100% standardization of OpenAPI Spec across dev projects -Dominos  Realized > 50% cost savings by switching from HP/Micro Focus
• #6: Validates your software system as well as interconnected sub-systems End-to-end testing is conducted and executed from the point of view of the end-user to replicate real user scenarios End-to-end testing is run after system testing What’s the difference: system testing validates your software system meets the specified functional requirements. It is performed in the context of your functional requirement specs, and it is run after integration testing and before acceptance testing.
• #7: Different components could include: interfaces, external databases, networks, or even other applications-–and their integration and communication to the workflow of the overarching application (and success of the product)
• #8: Software today is far from simple and only going to continue to get more complicated. In many cases it is a complex system, complete with a full-set of underlying components.
• #9: Confirm application health – End-to-end testing helps confirm that every part of your application is functioning like it should. From back-end to front-end interactions with the users on multiple systems. End-to-end testing also helps you validate that the application works on many environments Expand Test Coverage-–you will need to think about every environment that you should run your tests on. Think about your users, and test on those respective environments. Detect bugs—with end to end testing you’re testing at every sub-level and larger level, thereby testing at every iteration. When you do this you’ll be able to find bugs faster and fix issues earlier that you may not have found otherwise Reduce Testing Resources—this includes both time and money. If you find bugs faster, reduce the number of breakdowns, then end-to-end testing will also decrease your need to repeat tests and then the cost and time associated with that too
• #10: Horizontal testing In order to do horizontal testing you need your test environments for all of your systems to be set up in advanced Advantages: entails focusing tests on the user perspective, prevents issues from reaching production, ensures coverage for business logic requirements Vertical testing In order to complete vertical testing you’ll want to ensure each major stakeholder is involved in the project. It may be helpful to have a development strategy in place like behavior-drive development, test-drive development, or continuous testing. Advantages: High code coverage, faster test execution, more focused tests
• #14: You never want your user to notice any hiccups. You would expect the user to flow seamlessly from start to finish—through navigating the ecommerce process, finding the product they like, checking out, receiving an email, and getting started with the product.
• #15: An end user will rarely think of the back-end systems that are triggered each step along the way. So that is why it’s important to make sure the QA team is considering all steps that that make this process seamless. From what the user is interacting with and what calls need to be made. You need to make sure the page doesn’t 404, the UI behaves and looks as expected (the buttons work, the input fields are accepted etc.), that data can be entered, stored, encrypted, and saved as necessary.
• #16: So here comes the tricky part. You probably already have some workflows and tests put in place with your unit tests, UI tests, API tests, etc. and while it would be great to take a bottom-up approach basically meaning just merging all of your current testing practices to get the end-to-end testing coverage that you need however while this may help you skate by for a little bit you should really start from scratch and look at your user journey. When looking at your user journey this will help you figure out what workflows need to be defined. For this step it is important to think about the users journey. What will they do. Think about every detail and write it down. Define each of the workflows that a user may go thru noting that there is definitely. Not only just one workflow that a user of your application will go thru. From there you need to decompose your workflows into the smallest most atomic pieces you can while still making sure they are useful. From there you need to make some decisions. What do you want to automate, what browsers should you test, how are you capturing results, etc. This all sounds very familiar if you’ve ever joined us for previous webinars—specifically the webinar series on frameworks. But this top down approach will benefit you greatly in the long run.
• #18: Test Case Preparation—can be tracked with graphs visualizing the progress of your planned test cases that are being prepared against the number of ones that were originally planned Test Progress—the goal with this measurement is to analyze the progress and details of your completed tests week-by-week. Ideally, you’ll want to track the percentage of totall tests completed in addition to the ones that passed and failed. Defect Status—You will want to track the number of defects and focus on the percentage of defects that were opened and closed as well as the distribution of each based on their severity and priority Test Environment—You need to consider two key measurements—the amount of time allotted for running tests in a specific environment as well as the amount of time actually used while conducting your end-to-end tests in these environemnts. Example: count the total number of hours each environment is up and the total number of hours scheduled per day for testing
• #19: 1: Not including Environment in the End-to-end test – the User’s context is important, and includes OS, Browsers, devices, security permissions, and prior activity in the AUT 2: Focusing only on Functionality during an end to end test instead of the user’s actual experience                 This can be made worse through over-automating. 3: Executing End to end tests prior to functional tests leads to duplicate testing efforts.