Automated API testing comparison between Pytest, Playwright, Cypress, and Mocha(Supertest)

This is the second part of my previous article  — Automated API testing using Pytest library or Playwright? Part 1. 

I decided to make the second part extended to cover more testing frameworks.

I would like to make a Proof of Concept on the API testing framework with each of the mentioned libraries. 

This article is suites best for QA automation engineers dedicated to API testing and those who are starting their jorney, and are interested in seing the different approaches with different tools.

I suppose you are familiar with the basics of API testing, and I will not go deep into details.

Prerequisites:

You will need to have Node.js and Python installed.

The code used in the article can be found at: https://github.com/Valiantsin2021/API_tests_cypress_playwright_mocha_pytest

We will take a look at the implementation in code, easiness of setup, and speed of API test execution using:

- https://www.cypress.io/

- https://playwright.dev/

- https://mochajs.org/

- https://docs.pytest.org/

The application under test will be https://thinking-tester-contact-list.herokuapp.com

We will cover two CRUD scenarios:

1. Create -> get -> update -> get -> delete user
2. Create user-> create contact -> get contact -> update contact(PATCH) -> get contact -> update contact(PUT) -> get contact -> delete contact -> delete user

Example of single user CRUD:

There will be two groups of tests for each library to measure the running time and implementation in code:

• Single CRUD for user and contact scenarios (two test files)
• Loop (11 rounds) CRUD for user and contact scenarios (two test files)

Example of loop (11 rounds) CRUD for contact:

I will implement the same amount of tests in each library and measure the execution time, to see the advantages and drawbacks of each.

I will not cover the installation and setup of each library (it is described in the repository’s readme.md file), but will mention the specifics. There are no reporters set to have no saturation on the test run time.

Test data is generated using the Faker library via the custom-made dataBuilder utility (Node.js). Playwright and Pytest parse data previously seeded to JSON files. Cypress and Mocha leverage the inline data generation within the test code.

Pytest: 

I used pipenv to install all the necessary dependencies. For API requests I used the “requests” library.

Playwright:

I will use custom API class utility with a logger for debugging. I use the global setup module to leverage the dataFactory utility and seed the test data in fixture json files before the test run.

Cypress:

I use the cy-api plugin for debugging reasons and dataFactory utility to generate test data.

Mocha/Supertest:

Supertest is used to send the requests and Mocha/Chai for tests and assertions. Again the test data is generated using the dataBulder utility.

You can see all the dependencies installed for the JS testing libraries

Let’s start with Cypress.

Config file:

Tests in Cypress are the most verbose between all four libraries. <screen of test files> It is not making it worse, it is just a specific of the syntax. Its main aim — API testing is accomplished. Using the cy-api plugin from Filip Hric makes it quite easy to see the results of requests/responses and debug in Cypress Launchpad.

Due to the limitation of Cypress to run tests in parallel only on Cypress cloud, all the local test runs are sequential.

If we run a small set of tests, the execution time is reasonable and comparable to Playwright, but the more tests we run the more Cypress stays behind of its opponents. There is a beautiful plugin — cypress-split by Gleb Bahmutov to run Cypress tests in parallel for free on CI. There is no solution at the moment to parallelize the local test run, but I want to belive it will appear in a future.

The tests in loop for 11 users and 11 contacts CRUD scenarios showcased this very clearly.

Playwright:

Config file:

I would say the Playwright is the main opponent of Cypress nowadays. I personally love both tools. They are mature and have a great amount of unique features.

The Playwright setup uses the custom API handler class with the Logger utility to turn on and of the request/response logging on demand. These utility classes extend via the fixture to the test class. This is a drawback of the Playwright — we do not have out of the box way to visualize the API tests, but the same is applicable for Pytest, Mocha and Cypress as well. We have to use custom solutions for this.

I also have a global setup module to seed the generated data into the JSON files. This setup is run before all the tests starts, so we will have always the new data for our tests.

Playwright shows its power to run tests in parallel locally and on Ci out of the box and it can scale horizontally and vertically. You just provide the amount of workers via the command line or in the config file and the tests will be split between them and run simultaneously.

On the small set it is not providing the big advantage, but with the big sets — it ripes its fruits.

Mocha/Supertest:

Config file:

Mocha used to be a quite popular testing library, and it is famous for its customizability. The requests are made using the Supertest and the assertions are written the same way as in Cypress — using Chai library.

With the further evolvement of the JS frameworks Vitest and Jest are taking over the Mocha, but I personally like it for its simplicity and variety of reporters that the one could connect.

Mocha is also very fast, and we can use seamless integration with many npm libraries to extend its possibilities.

Pytest:

Config file:

Pytest is another loved library to perform not only the unit testing, but also the integration and UI tests. It is blazing fast and easy to setup. Using the pipenv we can encapsulate project dependencies in an easy way. Pytest has tons of plugins and extensions and is easy to write the test code. Agile parametrization and settings make it a powerful testing library. Even if we do not use the parallelization it is still increadibly fast.

Test execution:

So we have all four libraries setup with the same amount of tests to run. Two test files with single scenario and two test file with loop of 11 CRUD data-driven scenario.

To measure the test run I created the script in JavaScript.

All tests can be launched using the command:

node test_time.js        

Main script:

Results:

Here you can see the results for single scenario:

Here are the results of loop scenario:

Conclusion:

I do not pretend to be the ultimate truth.

I can also agree there is not a best approach to measure exacution time.

ou can spot the tendency from the screenshots, or try to run the tests yourself leveraging the code: https://github.com/Valiantsin2021/API_tests_cypress_playwright_mocha_pytest

Playwright showed itself as the fastest in a parallel run for the loop scenarios. When we talk about the individual CRUD — the winner is the Pytest, with the Mocha on second place. Cypress was struggling hard on the local run, but I believe it will change on CI with the use of cypress-split plugin.

I hope this small Proof of Concept was usefull and can provide you with some valuable insights on different testing libraries.

Feel free to write your opinions, I am always open for the feedback.

Happy testing!