The State of Testing 2017

Editor's Notes

• #3: Here are your hosts…. We will be recording the webinar, and for those of you in attendance, as well as for those who took the time to fill out the survey, we’ll be sending you the first copies of the full report at the end of the week.
• #4: Its going to be an interactive presentation with lot of opportunities for you to participate through polling and questions. 1 . Look at the survey methodology: add credibility to the data and know that we are presenting insights from real testers who work in the same space as majority of th audience 2. General testing trends: Some very important high level insights that we got, how many API endpoints a tester is using? 3. Automation insights ruled the data, insights around automation in industries and what is the relationship between automation and scripting abilities? 4. Processes like Agile and release cycles ,how often is software being released and what seems to be the right team size for successful Agile? 5. Test environments : we will look at what testers are using to test their frontend, browsers, mobile platofrms etc and API environments. 6. Finally we will take a look at the future of test automation, what our participants believe is the future of automation.
• #5: This survey was designed to establish benchmarks for the software testing industry regarding the ways in which teams are testing the APIs and Uis of their applications in 2017.   Collected a total of 5,200 responses with 3,400 completed via email to an internal database of developers, testers, managers and more. Joined forces with SmartBear Partner Cigniti, and influential industry group Ministry of Testing in Boston Promoted the survey in-product to grab customers actively using our test tools And to the larger testing community via google adwords and social media including facebook, twitter, and linkedin
• #7: Harsh: Participants in the survey got a chance to express themselves and provide open ended comments We tried to do some sentiment analysis and here are the results We asked: How do you think the software testing field will look in two years? Majority of the comments were around Automation tools -> shows the top of the mind thing for testers automation is the key term and is driving interest in processes and devops and impacting environments Development and coding is next on the top of mind of testers, more and more testing is being done in code, We found a link between coding proficiency and the extent of automation of tests
• #9: Harsh: Testing activities are not just focused on either frontend and backend but as per the results majority of the professionals are doing both at the same time. 63% of QA professionals do both, UI testing : A larger proportion of manual testers are doing UI testing than QA engineers and Software Developers Software developers are doing very less UI testing and a very large proportion of them are doing API testing Manual testers are doing the lowest amount of API testing, seems like API testing is more dependent on coding, scripting and increasingly being done a lot by developers Interesting to note that Only 10% of Manual testers are doing API testing and only 10% of Software Developers are doing UI testing, QA engineers have a large proportion of both UI and API testing Why do you think are more developers testing APIs than QA engineers and Manual testers? Why are manual testers less involved in API testing?
• #10: Harsh: Most of the teams are using unit testing! And some form of automation as well. Automated UI testing is used more than automated API testing. Service Virtualization seems to be a nascent concept and teams are not leveraging the value provide by service virtualization Interesting: Unit testing is prevalent amongst majority of the teams, Automation prevails in UI testing and slightly less so for APIs Testers are saying that unit testing is the most used methodology, what do you think are reasons for the popularity of this? Is there a learning here for teams not doing any unit testing? What about load testing 56% is it a high number? Service Virtualization is just 12% did you expect thart?
• #11: Harsh: API testing is getting more and more important and people understand the importance of testing APIs behind the frontend. We wanted to know how much time testers spend on APIs and what are they spending it on Most of the time is being spent be testers on functional testing their APIs, surprisingly very few people said that they are spending most of their time on Load testing which is less than the number of people who said they spend most time on creating mocks Interesting: that many people are spending maximum time on creating mocks for services, indicates that mocking is hard to do and how service virtualization solutions like ServiceV can help speed up the process Load testing seems to be behind mocking in terms of number of people who said they spend most time on? Endpoints in testing: what is the number of API endpoints people are using to test and it seems that most of them, i.e. 80% are using less than 50 endpoints for testing
• #13: Harsh: We asked people what %age of their tests are automated, also we asked them their company size. We found that 44% of the tests on an average are automated, larger companies automate more of their testing than smaller companies. Percentage of tests automated seems to be dependent on the size of the company, Interesting insights were around bigger companies having more tests automated and smaller companies will have lesser automation. What are the reasons that smaller companies have less exposure to automation? Bigger companies have
• #14: Harsh: A very interesting insight we found was around the scripting proficiency and automation we had a hypothesis that both have a relationship and we did find a strong correlation here. Higher the scripting proficiency, higher is the proportion of tests automated. Experts have 64% of their tests automated which is way higher than the average number of test automated i.e. 44% Beginner scripters said that only 11% of tests are automated More proficiency in scripting indicates more automation? How do we explain that? Is it scalable? How easy is it for teams to scale scripting based automation system?
• #15: We asked testers what industries they worked in and derived the percentage of testers in each industries who said that they have high proficiency in scripting As scripting skills correlates with the extent of automation in a team. So we decided to explore the industries where testers had the most scripting skills to assess their automation potential NGOs have the greatest proportion of testers who have proficient scripting skills, followed by pharma and government Finance and Healthcare are expected to have more automation, but what explains more scripting experts in NGO/Non Profit? What do you think about the potential of automation there?
• #16: Lets talk about specific languages, what languages are testers using while writing scripts and probably automating We decided to segregate the and analyze language skills by UI and API. Higher proportion of API testers use java, Python, Groovy PHP Higher proportion of UI testers use VB and C# What are the reasons that UI testers and API testers have different languages they are interested in? Poll: Do you script or do you use a tool for UI testing? Script Tool Higher number of API testers use Java, Python, Groovy and PHP Higher number of UI testers use C#, Visual Basic
• #17: We also got some insights about manual testing, seems like its not entirely dead yet, 56% of tests are still written manually Exploratory Testing was used by 45% of the testers, its mostly used in midsized companies, not very popular in small and very large size companies
• #19: We asked what’s the process that your team follows? Trends: https://trends.google.com/trends/explore?date=today%205-y&q=%2Fm%2F0867l,%2Fm%2F02zhbn Why are teams still doing waterfall? What does ‘It depend’ on the project means? Doo teams have to be flexible in choosing a process depending on the circumstance?
• #20: Here we segmented the results on the team size and this graph represents the %age of respondents in each segments that said that their teams follow Agile. 64% of the teams with members between 6-15 said that they were Agile. This number dips down to 56% for teams with 50 or more teams. Although 56% is a high proportion but 64% is higher, are teams finding immense success with the size of 6-15 in staying agile? Is this the sweet spot for the teams size?
• #21: Here again we looked at segments by team size and analyzed the proportion of teams that did Test Management Seems like bigger the team size the lesser the proportion of teams doing test management Poll: Are you happy with your test management solution? Yes/No
• #22: Harsh: We asked how often the participants release their software? And tried to see if there are any differences in release cycles between API projects and UI projects. UI exclusive testers more likely to release quarterly/yearly. More API projects get released in shorter release cycles multiple times a day or weekly APIs seems to have shorter release cycles and more UI is released over longer periods quarterly and yearly.?
• #24: This slide indicates the usage of different browsers by testers and the Mobile phone platforms used by testers Browser consumption, firefox more than IE? Why firefox not the biggest because of selenium? Blackberry? Fragmentation of platforms on android? Selenium angle? Poll: Do you write selenium scripts for multiple browsers? Ans: Yes/No
• #25: Liana: Which type of API do you test against most frequently? Pie chart with rounded numbers 6% are using primarily 3rd party API For unavailable APIs, 16% do nothing and 17% have no idea they have solutions available to emulate services. Poll: What do you do when you have to test against third party APIs? I doMocking API Virtualization I do Nothing
• #27: Explain slowly: This slide indicates the percentage of API testers and UI testers who think that their tests are fully automated now and will be automated 2 years in the future. So for example around 24% of API testers said that they have full automation now and 40% said that they expect to achieve full automation in 2 years from now. Similarly on the right side the percentage of QA engineers and QA managers amongst overall respondents who believed that their tests are fully automated now and will be automated 2 years in the future. API testers seem to be more confident about 100% automation in the future than UI testers, is it because they are seeing higher automation now and that makes them more optimistic? QA Engineers are more confident about 100% automation than are QA managers.
• #28: Definitely share with us your interpretation of the data with us. We are eager to hear from you.
• #29: Thanks for attending - we’ll be sending everyone the first copies of the full report at the end of the week. If you’re local to the greater Boston area, join us in September for our user conference that will consist of live, interactive product training classes and insights from industry experts and SmartBear customers. We’ll be dropping the link in the chat box for those of you interested in buying tickets.