Postbot: An overview of Postman’s Gen AI Assistant
If I were to ask you, the reader, a question: what is the latest technological advancement that has taken the whole world by storm, what would your response be? Anyone who is working in the tech industry and is up to date with it, their response would unarguably be Gen AI. Using Gen AI for one’s day-to-day tasks can lead to a significant boost in productivity. The reason is that generative AI takes care of menial tasks requiring little intellectual output but considerable manual effort. Consequently, a plethora of software development and testing tools are incorporating Gen AI capabilities.
Postman, the renowned API testing tool, has also integrated a Gen AI assistant i.e. Postbot to enhance its user experience. This article is aimed at providing the reader with an overview of Postbot’s capabilities and guidelines to integrate it into their testing process.
What is Postbot?
Postman’s official documentation states that Postbot is an ‘assistant for API workflows in Postman’. This phrase successfully captures the essence of Postbot, as it assists users in end-to-end API workflows in a variety of ways. These include generating documentation, test development, and troubleshooting APIs. Leveraging Postman’s capabilities can lead to a significant uptick in productivity. The purpose of this article is to explore these functionalities of Postbot and the optimal approaches to extract maximum benefit from them.
Getting Started
Access Points
There are two primary access points to launch Postbot. First, a bar at the bottom of the screen offers several options; clicking the ‘Postbot’ option will open the Postbot chat box. Second, within the Scripts tab of a request, an icon appears in the top right corner. Clicking this icon will also launch Postbot.
General Queries
So to kick things off, let’s start with the question: what is the primary purpose of a Gen AI chatbot? In essence, it is a conversational agent, which is designed to interact with an end-user and resolve their queries, just like an actual human customer support assistant would.
Therefore, the most feasible approach to interacting with postbot is to ask general queries regarding the APIs being tested in that particular collection. Bear in mind that Postbot’s scope is limited to a single request on Postman, so the query should be relevant to that particular API call.
For this article, I will be using a simple collection making calls to a pet store, for reference purposes. Now if someone else were to open this postman collection for the first time, they would not understand what is the process taking place through these calls. So they can use the the postbot to ask a question like ‘Can you explain what is happening in this request?’.
The postbot will generate an explanation like the one depicted in the image below.
Moreover, a user can also ask follow-up questions. For instance, there are multiple headers for an API call, and sometimes a user might have some confusion or ambiguity regarding one of them. So in such a situation, they can inquire the bot regarding the specific header.
Test Generation
One of Postbot’s most impressive use cases is extensive test generation for single requests as well as entire collections. There are multiple approaches to go about test generation using postbot and we will explore them one by one.
The most basic way to generate a test is to ask the bot to generate a test for verifying a specific field in the response. For example, consider a response to a post request, for ordering an animal from the pet store. The response has multiple fields like order ID, pet ID, quantity of pets, shipping date, order status, etc.
The most basic test to verify whether the process was successful would be to verify the status field in the response, to ensure that it has the value ‘placed’. Therefore, we can enter a prompt into the postbot stating ‘ Can you add a test to verify that the status field in the response has the value ‘placed’’.
The postbot will act accordingly and add a test in the ‘scripts’ section to verify the ‘status’ field in the API request’s response body.
Moreover, when the particular request is executed, the test results will also show up in their specific section in the bottom half of the postman user interface.
There is another approach for test generation using postbot, which is known as testing for response. In simple and plain language, testing for response is implemented to ensure that the API in question generates specific outputs against a request. Multiple parameters can be verified to ensure this; including response status codes, response body verification (verifying fields in the response body), text length, response time, etc. Postbot provides the user the capability to test an API’s response against these metrics.
Postbot, like many other Gen AI chatbots, has preconfigured interactive prompts, which are known as CTA i.e. call to action. These CTAs provide the option to perform a variety of actions and one of them is testing for response.
When the user selects the ‘test for response’ CTA, a subsequent menu of options is produced. This menu includes options like testing the response for data length, schema, content type, etc. In this particular example, if the option for content type is selected, the post bot will generate a test to verify the content type of the API response. Similarly, the other options for response verification can also be utilized as per the use cases relevant to the API request in question.
However, typing out prompts to specify each test is an extremely laborious and time-consuming exercise, especially in the case of requests with extensive responses. Fortunately, the postbot is intelligent enough to analyze a request by itself and generate all the requisite test cases for the request. Merely typing out a prompt along the lines of ‘add tests to this request’, is sufficient. Postbot will act on this and generate multiple test cases for the request in question.
Another capability that postbot possesses is to analyze, debug and rectify failing tests. For example, in this particular case, if I change the response code from 200 to 201 in the first case, it will cause that test to fail. If I enter a prompt, asking the postbot to correct the test for the response code, it will act accordingly and change the value of the response code to 200 in that test.
Test Generation for Entire Collections
It is understood that the above-mentioned approaches to test generation can be very hectic and time-consuming; particularly for extensive collections with hundreds of requests. Postbot offers a solution for this issue as well. Postbot can generate tests for the entirety of requests in a collection at once. The procedure is similar to that of test generation for a single request. Select the collection, and open the postbot chatbot. Selecting the ‘generate tests’ CTA will result in test generation for all the requests across an entire collection. The tests are displayed against each request in a tabular form.
An additional benefit of this format of visual depiction is that the user can analyze the tests for all the requests at one place, and alter them in accordance with the request. For example, while testing a particular request, its performance or response time is not that important compared to other metrics. You are aware of the API’s performance and are sure that the response time will always be more than the threshold specified in the test (e.g. 200ms). Either you can increase the threshold in the said request or remove the request altogether.
Visualizing Request Response
Postbots most prominent functionality apart from generating tests, is visualizing the responses for the API requests. The procedure is similar to that of test generation and fairly simple.
After selecting a request, the user can select the visualize response CTA from the postbot. They will be presented in the form of a menu with different visualization options i.e. bar chart, table and line graph. Upon selecting one of the options, the data in the response will be quantified and visualized accordingly.
For example, in the post request placing an order for an animal, the ID and quantity are depicted in the bar graph, since these are values in the response that can be quantified.
Documentation
Another one of postbot’s most prominent and highly effective functionalities is creating documentation for requests. Extensive documentation not only enhances knowledge transfer but also helps customer facing teams in debugging customer issues. However, it is also a fact that extensive documentation is a time-consuming ordeal.
This is where the postbot steps in. Simply clicking the ‘Add Documentation’ CTA against any request prompts the postbot to create comprehensive documentation for the said request.
In the example shared above, the postbot generated comprehensive documentation for a post request, covering aspects like request body, response body, purpose, request headers, etc. The user can even add more details to the documentation by sending follow-up prompts, as demonstrated in the example below.
Debugging Requests
A salient feature of postbot is its ability to debug failing API requests; similar to its ability to fix failing tests. This is an invaluable feature for any engineer engaging in API testing.
In the example below, there is some foreign and irrelevant text in the payload, which is causing the particular POST request to fail and return a 400 error.
Even though postbot might not immediately pinpoint the exact error, if it is pointed to a specific aspect of the request, i.e. the payload in the case, it can pinpoint and identify the error right away. This is a critical functionality and can prove to be of immense value when debugging complex requests with multiple headers and extensive payloads.
Limitations
Although postbot is very useful and effective, it comes with some limitations. These challenges and limitations are highlighted below:
1) Scope Limited to a Single Request
The biggest limitation of postbot at the moment is that its scope of operation is limited to a single request. The only exception in this case is test generation, where it can generate tests for an entire collection at once.
Otherwise, it cannot analyze any prompt outside the context of the request it is opened against. This makes it challenging to execute prompts against each request individually, particularly in the case of extensive collections.
2) Test Generation Prone to Errors
Another limitation that postbot manifests is that its tests are erroneous at times. Therefore, human oversight is necessary to ensure that not only the test cases are relevant, but their implementation in code is also correct.
For example, the date verification test originally generated by postbot was failing, as the regex expression to verify the date was incorrect.
However when the JavaScript implementation for the test was rectified, then the test passed.
Conclusion
To conclude the article, the postbot is an effective assistant despite some of its shortcomings. Therefore it should be utilized by developers and testers alike, who use Postman regularly. Doing so would lead to a significant uptick in their productivity. This article aims to acquaint the user with the basics of postbot, and how to leverage it to extract maximum benefit. The resources attached below offer a deeper insight into the product for those who are interested. Feel free to reach out in case on any queries. Happy Testing!
Resources
You can find Hamza's blog on our Medium, alongside many more. Happy reading!