Software testing and quality assurance
- 1. Software Testing & Quality Assurance Why using Specflow (Gherkin) and Selenium ? Benjamin Baumann @Zentiltoutou October 21st 2016
- 2. How much costs a bug? • 1962 NASA - Mariner 1 Omission of a hyphen in instructions. Auto destruct. $18M • 1996 Ariane 5 Integer Overflow on old legacy code that is not needed. No flight simulator to spare money. $370M for satellites alone
- 4. How much costs a bug? • 2009 Toyota Lag in the anti-lock-brake system. Recall 9M cars. $3G estimated • 2012 Knight Capital Group Code deployed to all but one server. This server sent out 4M+ wrong stock orders in one hour. $440M
- 6. But we’re no Spacecraft or trading firm! • 2015 Eurosport Player V6 Video lags, fps drop, overheating PC, angry clients. Due to incompatibility between technology and requirements. Detected by beta testing but gone live before the test results
- 7. Ok. Let’s test! White-box? Black-box? • White-box testing Tests the internal structure or workings of a program : • Unit tests • Functional testing • Integration testing • Regression testing • Load & Stress testing • Sanity & Smoke testing • Black-box testing Software is a black box. Tester knows nothing about how it works. • Acceptance testing • Beta testing
- 8. Unit tests are idiots.
- 9. Unit tests are idiots.
- 10. Unit tests are idiots but legion! • Think about code coverage • Assess code quality, not software’s • Think about edge cases, error handling
- 11. Unit tests are idiots, legion and hard to maintain • Each code change/writing begin with test change/writing • « I’m a developer, these tests are idiots! Why do I have to write them? » • « The tests failed, such a pain. I’ll just comment them out »
- 12. Break #1 : That’s not me. All internet is saying it!
- 13. That’s not me. All internet is saying it!
- 14. That’s not me. All internet is saying it!
- 15. That’s not me. All internet is saying it!
- 16. Still, I want to do unit testing • Please, do unit testing only if : • You have a decent number of developers in your team • You have time to do it and to maintain it • You can run them and validate them automatically (through CI) • You run a business critical project • You have detailled technical specs • Coding unit test is still coding, treat your tests like your code • Don’t write the code AND the tests, make someone else write the tests. • Golden rule : « Each unit test case should be very limited in scope »
- 17. Ok I have a ton of unit tests. Am I OK?
- 18. Ok I have a ton of unit tests. Am I OK?
- 19. Focus on functional,regression and acceptance tests • Testing software features above code methods • Less detailed, less to do, less to maintain • Focus on testing the requirements/specs • Safeguard for developers
- 20. Ok let’s test a feature ! • Describe the feature : Who? What? Why? • « In order to watch the player As a Eurosport Player user I want to be able to connect» • Help describing the need • Feature-centric • Technology agnostic • Need for testing scenarios
- 21. Ok let’s write a scenario ! Given I go to the Eurosport Player home page And I open the connection modal And I enter my credentials When I submit the form Then I must be logged in White-box? Black-box?
- 22. Ok let’s write a scenario! Given I go to the Eurosport Player home page And I open the connection modal And I enter my credentials When I submit the form Then I must be logged in White-box! Describing a feature should be blackboxed.
- 23. Ok let’s write a better scenario! Given I go to the Eurosport Player When I login Then I must be able to watch Eurosport Player No tech. No design stuff. Just the feature. Feature does not change when code is refactored. You should rewrite the code behind, but not the test scenario
- 24. Break #2: Understanding others
- 25. Break #2: Understanding others • « Hey, this is not the behaviour I asked for !», « But that’s what you told me in your mail… », « You did not understand, that’s not what I wanted » • « Domain, Network, AD, I mix up all these, it’s gibberish to me » • Solutions • Learn the other language so that you understand them • Everyone learns a new language so that we understand each other (AWS) • Don’t go into details, just do high level • Speak like you’d speak to the average man in the street
- 26. What about Gherkin? • We’ve already used Gherkin • Gherkin is the language that Specflow understands. It is a Business Readable, Domain Specific Language that lets you describe software’s behaviour without detailing how that behaviour is implemented. • Gherkin serves two purposes — documentation and automated tests. The third is a bonus feature — when it yells in red it’s talking to you, telling you what code you should write.
- 27. Documentation by Gherkin • List features • List testing scenarios • Readable by EVERYONE • Hard to write, easy to understand • A heaven for maintenance • http://qa.sti.eurosport.intra/Home/SpecflowList?path=d%3A%5Csites %5CQA%5CProjects%5CEurosport%20Player%20PROD%5CScenarios& projectName=Eurosport%20Player%20PROD
- 28. Automated tests with Specflow • Each scenario line is linked to a code method • Pass scenario data to the method • Scenario Context (browser, language, environment…) • Runnable in classic test runners (NUnit, MsTest …) • Explicit errors • When failed, need to go deeper with unit tests or by hand
- 29. And what about Selenium? • Selenium WebDriver is a library that can drive a web browser • Method on the webdriver results on « real user » action • Let’ code!
- 30. Merci!