SwissQ Testing Trends & Benchmarks 2012 (Englisch)
2 likes1,074 views
The SwissQ Testing Trends & Benchmarks 2012 report offers insights into software testing trends, highlighting the increasing adoption of Agile methodologies, with 84.5% of organizations using Scrum. It identifies key challenges in software testing, such as poor requirements quality and late tester involvement, while noting that investments in test automation are rising, despite a significant portion of companies still lacking adequate automation. Additionally, the report emphasizes the need for better interdisciplinary collaboration among team members, particularly in requirements engineering and Agile practices.
SwissQ Testing Trends & Benchmarks 2012 (Englisch)
- 1. SwissQ Testing Trends & Benchmarks Switzerland 2012 Software Testing Trends Switzerland 2012. Where are we now – where are we going to?
- 2. TABLE OF CONTENTS SwissQ Testing Trends & Benchmarks 2012 2 3 EDITORIAL 4 TRENDWAVE 2012 6 KEY MESSAGES 7 PROJECTS 8 MATURITY 9 ORGANISATION 11 PROCEDURES 12 TESTING TOOLS 13 FRAME OF SURVEY 14 REQUIREMENTS 15 AGILE
- 3. EDITORIAL SwissQ Testing Trends & Benchmarks 2012 3 For the fourth time in a row, SwissQ questioned the Swiss IT community about current trends and benchmarks. Not only did we receive overwhelming feedback from over 300 participants, we also personally interviewed many IT decision makers from various companies, segments, and regions about the current testing trends. The outcome is a representative overview of the most important trends. For the first time this year we are not only presenting a trends and benchmarking analysis of Software Testing, but are also taking a look at Requirements Engineering and Agile. The corresponding reports will be published separately to highlight their importance. The group of participants mainly included test managers and line managers, but The same as last year, SwissQ offers companies the opportunity to present the also test engineers and project leaders working in the area of IT, finance, insu- SwissQ Testing Trends & Benchmarks to their employees. The target audience rance, manufacturing, or in public companies. For the first time this year we put consists of members of the testing community, software engineers, project leaders, a special focus on Agile and Requirements Engineering and have thus enhanced requirements engineers, and the management. The presentation allows you to the group of participants accordingly. A quarter of the participants of this survey, take an outside look at the current changes, as well as to boost the motivation for therefore, work in Requirements Engineering. In the end, these 300+ question- and significance of testing, and profit from ideas for further developments. naires served as the basis for the analysis and interpretation of the current state in the year 2012 and the future development. Having said this, we hope that the present Trends & Benchmarks will inspire you to actively pursue new challenges and take take the actions that best suit your The benchmarks depicted in a multitude of informative charts and diagrams form business. the backbone of this report and allow you to position your company in comparison to others. As before, we use the SwissQ Trend Wave® to show the evolution of the main topics. It shows in four phases how single trends will most likely develop over time and in turn allows you to appraise the influence of the trends on your business.
- 4. TRENDWAVE 2012 SwissQ Testing Trends & Benchmarks 2012 4 INTRODUCTION GROWTH MATURITY DECLINE PRIORITY Automated Test Process Improvement Regression Tests ALM Tools SCRUM Agile Testing Regression Tests Load and Performance Tests Central Test Organisation Explorative Testing Risk Based Testing ISTQB FL & AL Level TDD Security & Penetration Testing Outsourcing ISTQB Expert Level Session Based Testing Mobile Testing Cloud Testing ATDD TIME INTRODUCTION – This topic has been GROWTH – This topic is more and MATURITY – Most companies are DECLINE – The topic has already identified and some companies are more accepted and many companies working on the implementation been implemented by most of the deploying initial implementations. are considering it. The first tools are or have already completed it. companies, with the exception of However, it cannot be foreseen being developed and consultancy The knowledge of this topic is individual latecomers. Often, there whether this trend will positively firms offer services for the same. often widespread, resulting in is no more added value in acquiring advance and whether testing will Often risks are associated due to sub-topics being raised. further knowledge in these areas, be considerably influenced. limited implementation experience. since it will become obsolete shortly.
- 5. TRENDWAVE 2012 SwissQ Testing Trends & Benchmarks 2012 5 The aim of the trendwave is to show what topics will influence the testing discipline as well as the people and the organisations dealing with it. It thus helps readers to specifically prepare for, and actively participate in these changes. The technologies that changed the consumer market in the last couple of years – The current focus remains on two areas that have continued to gain significance smart phones and tablets, social media, cloud, and highly topical, voice control over the past few years, one of them being Agile Testing as most of the partici- – are increasingly being used in the business world as well. The new information pants of this survey are at least partly working with agile methods. The second and communication channels accompanying this development lead to higher area is Requirements Engineering which is also directly involved in one of the consumer expectations as the following example illustrates: biggest challenges in testing: the generally poor quality of requirements. Be it coincidence or fate, the parallel development of these two subjects can only be of advantage to us. During the après-ski break, we started a heated argument about the need of a separate ski theft insurance. I got my smart phone and opened the app of my We can therefore assume that Mobile Testing and especially Cloud Testing for many insurance company to check my current coverage. I was startled to realize that businesses and software testers will only appear on the radar once the biggest I wasn‘t insured against ski theft! I immediately asked Siri to find a provider and obstacles in the transition to Agile Testing and the improvement of the quality of took out a policy online. I was lucky I did that because I just saw a guy in a yellow requirements are overcome. Furthermore, test automation – still not on a ski suit taking off with my skis! satisfying level – also remains in the focus. Still, it is conceivable that the “mobilization“ of IT will increase rapidly in the Such seemingly utopian scenarios are on the verge of their break through today. near future, so the question to pose is: shouldn‘t we jump on the bandwagon They are not only challenging business analysts and software engineers, but tes- right now before catching up will be too cost-intensive and time-consuming? ting as well will become a more complex and costly endeavour. Many companies have recognized these trends and are already including them in their current projects and products, others are gaining experience and building up more knowledge. Most businesses though still think that it is too early to deal with the subject but should be careful not to miss the opportunity to catch up on it.
- 6. KEY MESSAGES SwissQ Testing Trends & Benchmarks 2012 6 1 More than half of the companies apply agile development practices. Thereby 84.5 % of the respondents use Scrum as their preferred agile method. 2 New technology and process driven processes like mobile testing, agile testing or session based test management are (still) getting little attention. 3 The focus of education and training has shifted towards requirements engineering and agile, closely followed by the ISTQB advanced level certifications. 4 While 40% of the respondents rate the maturity level of the testing process as good or excellent, 75 % of the respondents rate the maturity level of the requirements engineering 5 Not surprisingly, poor requirements are mentioned as the biggest challenges in software testing, Followed by late involvement of the testers. 6 The interdisciplinary collabora- tion between business analysts, software engineers, and testers gained further importance. But the significant mind shift hasn’t as medium or less. happened yet. 7 The most investments are made in test automation or the creation and extension of regression tests. In contrast there are just marginal investments in hiring of new 8 Even though 80% of all participants automate their functional tests, almost half of them automated less than 10 % of all functional tests. This is a step forward though, as 9 The trend towards integrated tool solutions has increased, parti- cularly thanks to HP ALM and partly MS TFS. Tools covering only one discipline find themselves testing employees. last year almost 2/3 of the in a difficult position. participants had automated less than 10 %.
- 7. PROJECTS SwissQ Testing Trends & Benchmarks 2012 7 Project Type 70 % of all projects are new developments or updates of existing software. >50% of the respondents describe the starting situation of their 12 % projects as satisfying or insufficient related to 8% New development Estimation 39 % Update of existing software Planning 10 % Migration Definition of requirements Implementation of standard Realistic expectations software Operation, support, maintainance, 31 % re-design, ... Project Outcome Just over a third of all projects are finished with the expected functionality, within the expected timeframe and within budget. Project Size (in Swiss Francs) 40 % 51 % 35.1 % 30 % 40 % 39.2 % 39.2 % 25.1 % 20 % 18.1 % 17.5 % 20 % 10 % 4.1 % 10.8 % 0 % Project Proj. finished Proj. finished Project Project finished in time, with budget with major extended / stopped 0 % budget and and / or time functional rescheduled up to 1 Mio up to 20 Mio more than 20 Mio functionality overruns changes
- 8. MATURITY SwissQ Testing Trends & Benchmarks 2012 8 Maturity of Test Process Reputation of Testing In average the respondents rate the maturity of their companies testing process as medium or good. This rating has remained constant in the last years. It’s a contrast to the maturity in requirements engineering (RE) Testing is strategic for 18 % which is rated substantially worse. company success 50 % Testing is important for 49 % reliable software 40 % Testing is a necessary evil 19 % Testing has low priority 13 % 30 % Testing RE Testing is unnecessary, we can 2 % save the costs for testing 20 % 0 % 10 % 20 % 30 % 40 % 50 % 10 % 0 % The Biggest Challenges ea k ea k um od nt w W i Go lle ry ed ce Poor Availability Ve M Ex Requirements of test environment Testers are involved 63 % 30 % Auto- too late mation 38 % 40% 75% 33 % Not enough of the respondents rate the of the respondents rate budget / resources maturity of the testing process the maturity of the as good or excellent. requirements engineering 30 % as medium or less.
- 9. ORGANISATION SwissQ Testing Trends & Benchmarks 2012 9 Who Performs Testing Activities Education The focus of the employees training is moving from ISTQB certification Test Case Design – which most of them already have – to the topics of requirements engineering and agile. I already have it It is planned In the not too distant future IT Tester ISTQB Foundation Level 36 % Req. Engineer / ISTQB Advanced Level Business Test Manager Developer Analyst IT Tester End User ISTQB Advanced Level 21 % 16 % Test Analyst 19 % Developer ISTQB Advanced Level 7.4 % Technical Test Analyst IREB® CPRE Foundation Level Project Management (IPMA, PMI, ...) Test Execution Certified Scrum Master Agile Testing Certified Product Owner Certified IT Process and Quality Manager IT Tester Agile Requirements Engineering 34 % End User 26% ISTQB Expert Level Developer Req. IREB CPRE Advanced Level IT Tester Engineer/BA Elicitation & Consolidation 17% 14% Developer IIBA CBAP (Certified Business 9 % Analysis Professional) IREB CPRE Advanced Level Requirements Modeling 0 % 20 % 40 % 60 % 80 % 100 %
- 10. ORGANISATION SwissQ Testing Trends & Benchmarks 2012 10 Use of External Resources Testing Effort The use of external resources in testing slightly increased compared The variance of the testing effort is still wide. By average the testing effort in to 2011. The external resources are mostly used in-house. proportion to the total effort is about 15 % to 20%. In proportion to the deve- Outsourced testing stagnates compared to Managed Service lopment effort this value is more than 50 % (not represented in the graph). (e.g. automated regression tests) which have expanded. 30 % 5.9 % 30 % 10,4 % No external resources 33.8 % Acquisition of external resources 20 % 21 % (Insourcing) 19 % Acquisition as a service 17 % (Managed Service) 50 % Outsourcing to external resources 10 % 8 % 4 % 1 % 0 % < 5 % 5 – 10 % 10 – 15 % 15 – 20 % 20 – 30 % 30 – 50 % above Testing effort in proportion to total effort Investments Investments are Investments remain Investments are increasing constant decreasing Test automation Test environments Outsourcing testing activities Test management tools Hiring new testing employees Creation and extension of Test data management regression tests Training and education Structured testing process / methods
- 11. PROCEDURES SwissQ Testing Trends Benchmarks 2012 11 Software Development Process Implementation of Agile Methods More than half of the companies use agile processes. Many of them are using a combination of different development processes e.g. agile Drivers of Agile Methods in combination with waterfall. Agile processes are mostly promoted by the directly involved people. 60 % 39.2 % 52.1 % Agile 40 % 39.4 % 38.0 % 36.6 % 51 % Waterfall 10 % 19.8 % 40 % 0 % Developer Departement Head Project CEO Iterative mgr. / Team development Manager Hermes lead 22 % RUP 12 % 16 % Reasons for Agile Methods 84.5% Better handling Accelerate Time-to-Market of the respondents use Scrum as their favorite agile method. of changing priorities Productivity SwissQ Agile Trends Benchmarks 2012 Collaboration between There are many additional charts and details in the IT and business SwissQ Agile Trends Benchmarks Report 2012.
- 12. TESTING TOOLS SwissQ Testing Trends Benchmarks 2012 12 Test Management Tools Test Automation More than 60 % of the respondents work with HP QC (since version 11 the name is changed to ALM). Many of them are working with more than one Tools used 80% tool, mostly in combination with MS Excel or Word. MS Test Manager has a The market leader by far is HP Quick Test Professional, followed growing share, due to its tight integration with MS Team Foundation Server. by proprietary tools. All in all the market is highly fragmented. Almost 30 tools were mentioned. of the respondents automate their functional tests. HP QC / ALM 62 % Selenium MS Office 44 % Tosca Canoo Webtest Proprietary Tools Ranorex Lisa 48% of the respondents have Open Source 20 % Quick Test Professional automated less than 10 % of their functional tests. Rational Functional Tester In 2010, the value was 60 %. QF Test Others Proprietary developement 8 % MS TFS 2010 Test Manager 5 % Cost Savings by Test Automation Most of the respondents expect low savings by test automation. Inflectra Spira 2 % In one third of the cases giving a statement to this question was not possible. IBM Rational TestManager 2 % 33.3 % No tool 1 % 22.6 % 23.7 % 0 % 20 % 40 % 60 % 10.2 % 2.8 % 7.3 % Costs up to 10 % up to 20 % up to 50 % up to 80 % No increased statement possible
- 13. FRAME OF SURVEY SwissQ Testing Trends Benchmarks 2012 13 Industrial Sector Responsibilities More than 60 % of the respondents work either in the IT or in the More than 50% of the respondents describe their job with more than financial sector. Compared to the last years their proportion has decreased, one role. Especially test managers don’t work 100% as test managers, demonstrating that the subject has arrived in other industries too. but also take responsibility for other roles. IT 36.1 % 30 % Finance, Insurance 28.4 % Manufacturing 7.4 % Public and semi-public companies 7.4 % 20 % Traffic and Transportation 5.6 % Telecommunication 4.0 % MedTech 3.7 % 10 % Others 7.4 % 0 % 10 % 20 % 30 % 40 % 0 % IT Employees ge r na ana er r A ge / B nee age r r Te st er na ge r ne er a e n gi a n a ngi A bit more than half of the respondents work in companies with more t M n M gin st E ct M s M re E es io n e e nt a than 500 IT employees. T is ts E T oj e iv Pr m ftw / D men u ire So t q en uire Re rtm eq pa R 2001– ... 33.0 % De 501 – 2000 17.6 % 60% 33% 251 – 500 13.6 % 51 – 250 15.4 % 11 – 50 14.2 % of the respondents mainly of the respondents are 1 – 10 6.2 % work in projects. line managers. 0 % 5 % 10 % 15 % 20 % 25 % 30 % 35 %
- 14. SWISSQ REQUIREMENTS ENGINEERING SwissQ Testing Trends Benchmarks 2012 14 TRENDS BENCHMARKS REPORT 2012 Do you want to know more? You can download the detailed report with further analyses about requirements engineering from www.SwissQ.it SwissQ Requirements Engineering Trends Benchmarks 2012 There are many additional charts and details in the SwissQ Requirements Engineering Trends Benchmarks Report 2012. Future Investments Investments Investments remain Investments increase constant decrease Education and Training for Employees Better Cooperation 33 % 54 % 13 % 80% 36% of the projects check their do not check requirements of Business and IT 33 % 53 % 14 % requirements for operational for their need. Standardisation of the accuracy, implementability, internal RE-Processes 25 % 61 % 14 % and completeness. Elaboration / Definition of the RE-Role 24 % 60 % 16 % Development of Templates and Guidelines 22 % 61 % 17 % 2/3 Hiring new RE-Employees 22 % 55 % 23 % Establishing specific RE Tools 21 % 64 % 15 % of the respondents Establishing internal allocate less than 1 day RE-Divisions/-Departments 17 % 63 % 20 % for stakeholder analysis. Outsourcing RE-Activities 11 % 48 % 41 % 0 % 20 % 40 % 60 % 80 % 100 %
- 15. SWISSQ AGILE TRENDS BENCHMARKS REPORT 2012 SwissQ Testing Trends Benchmarks 2012 15 Do you want to know more? You can download the detailed report with further analyses about agile trends from www.SwissQ.it SwissQ Agile Trends Benchmarks 2012 There are many additional charts and details in the SwissQ Agile Trends Benchmarks Report 2012. Main Reasons for the Failure of Agile Projects Only 40% Lacking Experience with agile Methods 52 % Corporate Philosophy is not connectable with agile Values 45 % are content with the implementation of their External Pressure to follow a agile approach. traditional Approach 41 % Lacking Support of the Management 38 % Lacking / Insufficient Training / Coaching 36 % 1/3 Lacking Interconnections between organisational units 35 % of the respondents have Lacking Team Motivation 22 % 2 or more years experience with agile methods. Others 12 % 0 % 10 % 20 % 30 % 40 % 50 % 60 %
- 16. ABOUT US SwissQ supports its clients in the development and implementation of IT-solutions and assures that the end users get the functionality they really need. This is achieved by unambiguously determining requirements and risk-based testing the implementation. Our vision is to improve the added value of IT through requirements management and software testing. Along with providing high-quality services, we pursue this vision by establishing independent platforms, like the Swiss Testing Day and the Swiss Requirements Day, which facilitate the exchange of know-how and experiences. In addition to that we help bright minds to expand their knowledge in our trainings. © by SwissQ Consulting AG | Stadthaus-Quai 15 | Switzerland-8001 Zürich www.SwissQ.it | info@SwissQ.it | Phone +41 43 288 88 40 | Fax +41 43 288 88 39 Twitter: @SwissQ | Facebook: swissqconsulting