Streamlining End-to-End Testing Automation
0 likes331 views
The document discusses the complexities and challenges of end-to-end test automation in distributed teams, particularly in hybrid working environments with varied system setups. It outlines solutions for consistently setting up test environments, managing dependencies, and executing tests across multiple platforms using the 'teswiz' framework, including automation for CI/CD pipelines. Additionally, it emphasizes the importance of establishing a 'hard gate' for tests to ensure product quality and integrates AI for enhanced validation and reporting.
Streamlining End-to-End Testing Automation
- 1. @BagmarAnand Streamlining End-to-End Test Automation Anand Bagmar Software Quality Evangelist
- 3. Ground Reality Distributed teams Hybrid working Different network setup and speeds Many teams (>100) Mac, Windows and Linux laptops Different Software versions Certificates, Policies and multiple VPNs Inconsistent Developer & SDET experience Test execution environment setup is tedious System Tests (e2e & component UI): teswiz (Appium, Selenium, Applitools, JDK 17) • Emulator/Simulator setup (Android SDK, xcode) API & API Workflow tests: karate Contract tests: Specmatic Unit tests, Sonar code quality checks Test automation toolset Many environments Test data Branch Configuring appropriate test execution in build and release pipelines Complex path to production ADO agents: Windows Server & Linux agents Firewall restrictions download dependencies Direct access prohibited to CI agents Multiple node & JDK versions Connectivity issues to application-under-test No browsers/devices on CI agents CI execution @BagmarAnand
- 5. Getting a simple automated test to run consistently for all Developers and Testers and in CI (ADO) is painful! Setup Execution (full or specific tests) @BagmarAnand
- 6. Challenges of End-2-End Test Automation Ensuring Test Environment Consistency Coordinated Test Execution Test setup & execution on CI Agents @BagmarAnand
- 9. • Setup important applications on Mac • https://gist.github.com/anandbagmar/92b9f92298b1e17fa32c3404ad115 871 • Script to setup Android SDK on Mac • https://github.com/anandbagmar/AppiumJavaSample/blob/master/setup AndroidSDK.sh • Script to setup Android SDK on Linux • https://github.com/anandbagmar/AppiumJavaSample/blob/master/setup _linux.sh Test Authoring Environment Setup @BagmarAnand
- 10. • Node script to install all dependencies (for system tests): • https://github.com/znsio/getting-started-with- teswiz/blob/main/package.json • npm install – and you are ready! Test Execution Environment Setup @BagmarAnand
- 12. • Setup should be simple – Ex: • git pull • ./gradlew build • No code change required for • Running tests against any environment (local, dev, qa, staging, prod, etc.) • Test data and environment configurations are separately maintained • Running all or subset of tests • Tests should run from command-line Test Automation Framework Criteria @BagmarAnand
- 13. Your Ultimate Open-Source Solution to Automate Real-User Scenarios! @BagmarAnand
- 15. Run tests from CLI Test Authoring Execution Setup 1 2 2 3 4 5 6 6 Execution Reports CI Tool Feature coverage @BagmarAnand 16
- 16. • Web browsers • Mobile-web browsers • Android apps • iOS apps • Windows desktop apps • Electron apps Platform support @BagmarAnand
- 17. • Open source framework to automate real-user scenarios • Multi-user • Multi-device • Multi-app • Setup a HARD-GATE for your functional tests! Unique capabilities of teswiz @BagmarAnand
- 18. • Cloud device farm integrations • Applitools AI for validations • Comprehensive reports with trend analysis, feature coverage, failure analysis using AI-ML • CLI • Configurable Unique capabilities of teswiz Defaults Property files Environment Variables @BagmarAnand
- 21. Use the right node version @BagmarAnand
- 22. Use the right node version @BagmarAnand
- 24. • For System Tests, the artifact (apk/app) could have been generated from another pipeline • This artifact needs to be available in local or cloud device before tests can start execution Script for downloading artifacts – Why? @BagmarAnand
- 25. • Understand the CI tool APIs • Script downloads the android/iOS artifact for: • Specific branch • Latest successful build, or a specific build number Script for downloading artifacts @BagmarAnand
- 26. • Script uploads the android/iOS artifact to your device farm OR • Teswiz can upload it automatically for you Script for downloading artifacts – Bonus! @BagmarAnand
- 28. • Understand what dependencies in your framework need proxy information. Ex: • Gradle/maven • Downloading newer version of browser drivers • Any external connectivity • Framework should be configurable to pass this at test execution time. Ex: • No proxy required from local laptop execution • Proxy required when running tests from CI Proxy Handling @BagmarAnand
- 29. @BagmarAnand
- 31. Downloading dependencies - Uber jar #6 @BagmarAnand
- 32. • To reduce the number of dependencies to be downloaded, teswiz is built as a uber jar. • Specify only “teswiz” as a dependency in your test framework Uber jar @BagmarAnand
- 33. • Run as a java process • ./gradlew run Uber jar @BagmarAnand
- 35. • CI agents may not have browsers installed • The installed browser may be an older version Run browser in docker – Why? @BagmarAnand
- 36. • Should allow support for any os/architecture • Should allow choosing the browser (ex: firefox, chrome, etc.) • Should allow starting the containers with specific project names and dynamic ports to prevent conflict with multiple test executions • Support specifying proxy information • Can be used on local laptops as well as in CI executions Run browser in docker @BagmarAnand https://github.com/znsio/teswiz/blob/main/dockerContainers.sh https://github.com/znsio/teswiz/blob/main/docker-compose-v3.yml
- 37. Template for running tests in build pipeline #8 @BagmarAnand
- 39. • Create templates • Reuse with appropriate configuration parameters Running tests in Build Pipeline @BagmarAnand
- 40. @BagmarAnand
- 41. Task Groups for running tests in release pipeline #9 @BagmarAnand
- 43. • Create Task Groups • Include in each relevant stage of Release pipeline Running tests in Release Pipeline @BagmarAnand
- 44. @BagmarAnand
- 45. @BagmarAnand
- 46. @BagmarAnand
- 47. @BagmarAnand
- 48. @BagmarAnand
- 49. @BagmarAnand
- 50. Hard Gate Make your tests valuable! #10 @BagmarAnand
- 51. @BagmarAnand What is a Hard Gate! Why is it required? https://github.com/znsio/teswiz/blob/main/docs/HardGate.md • Automated tests should allow you to take decisions on product quality
- 52. @BagmarAnand What is a Hard Gate! Why is it required? https://github.com/znsio/teswiz/blob/main/docs/HardGate.md • For every test execution cycle: • Passing tests are expected to pass • Known Failing tests are supposed to fail, unless: • The product (bug) is fixed, OR • The test is fixed/updated If either criteria is not met, the build should fail!
- 53. @BagmarAnand Hard Gate - Make your tests valuable! https://github.com/znsio/teswiz/blob/main/docs/HardGate.md Build passes if Hard Gate criteria is met. Build fails if • one or more passing tests have failed, or, • one or more failing tests have passed
- 54. @BagmarAnand Functional (e2e) Automation as Hard Gate! https://github.com/znsio/teswiz/blob/main/docs/HardGate.md
- 55. Applitools Visual AI for validations #11 @BagmarAnand
- 57. 90% less code to write & maintain with infinite coverage. WITH APPLITOOLS AI Every Element Is Validated They Look & Work Perfect Use Applitools Ultrafast Grid (UFG) - Test is simpler – one call to Applitools (eyes.checkWindow()) validates the full screen - Run the test once - Get results from all browsers automatically - Less test data - No additional load on the application environment @BagmarAnand
- 58. • Works for all platforms • Native & hybrid apps – android, iOS • Web browsers • Desktop applications • Electron applications • Seamless scaling using Applitools Ultrafast Grid @BagmarAnand Applitools Visual AI
- 59. Specify as many browsers with viewports and devices as required for validation You do not need to do cross-browser validation at the end anymore! @BagmarAnand AI-powered Cross Browser Test Automation
- 60. reportportal as a Central reporting server #12 @BagmarAnand
- 61. @BagmarAnand
- 62. A central reporting server for your organization @BagmarAnand
- 63. Test Execution Real-time status • See progress of launches currently in progress • Can also see details of tests that are currently running, till the point of execution @BagmarAnand
- 64. Test Execution Details – Device farm report link & Device logs • The link to the device farm test execution dashboard is available in the result • teswiz attaches browser logs/device logs automatically to the result in ReportPortal @BagmarAnand
- 65. Test Execution Details – with screenshots • The test result includes screenshots as captured by the test @BagmarAnand
- 66. Test Execution Details – Applitools Visual AI Validation Results • The test result includes the status of Applitools Visual AI validation • Link to the Applitools dashboard is available in the result @BagmarAnand
- 67. Test Execution Trend Analysis • Each test shows the trend of its execution – giving an indication of (in)stability @BagmarAnand
- 68. Test Results – Next Steps • On investigation of the failed tests, mark the failures with appropriate reasons (as configured) @BagmarAnand
- 69. Auto-analysis of failed tests Analyse the failure reasons by Auto-Analyzer based on Machine Learning @BagmarAnand
- 70. Auto Analysis of Test Failures • Why waste time marking the test failed for the same reason as last time? • ReportPortal can do this automatically for you with the Auto Analysis and Pattern Analysis feature @BagmarAnand
- 71. Auto Analysis of Test Failures @BagmarAnand
- 72. Test Result Visualization Configure simple and understandable reports • Create as many dashboards as relevant for the team • Dashboards may be for different persona/role, giving appropriate information @BagmarAnand
- 73. • Teswiz and karate test frameworks can automatically upload test results to your reportportal server • sendToReportPortal: • https://github.com/znsio/sendToReportPortal/blob/main/importRes ultsAndUpdateAttributes.sh • Can upload junit test results generated by any type of tests to reportportal with relevant test execution metadata reportportal.io @BagmarAnand
- 74. Challenges • Ensuring Test Environment Consistency • Coordinated Test Execution • Test setup & execution on CI Agents Solutions Consistent environment setup Test Automation Framework support Node setup Script for downloading artifacts Proxy Handling Downloading dependencies - Uber Jar Browsers in docker Template for build pipelines Task groups for release pipelines Hard Gate AI for validations Central reporting server Summary @BagmarAnand