Modern Testing Strategies for Evolving Ecosystems
Download as PPTX, PDF0 likes157 views
The document outlines a modern testing strategy for evolving software ecosystems, emphasizing the importance of various testing methods including unit testing, integration testing, and exploratory testing. It highlights trade-offs like maintenance costs, execution time, and their impact on design while discussing strategies to ensure system correctness and effective coverage. The document also addresses advanced techniques such as property-based testing, stress testing, and behavior-driven development, aiming to structure and improve testing processes within complex systems.
Modern Testing Strategies for Evolving Ecosystems
- 1. Modern Testing Strategy For Evolving Ecosystems @hundredmondays
- 2. Why tho? ● Ability to prove system correctness ● Reasonable maintenance cost ● Low cognitive burden of writing tests ● Rapid feedback from test execution ● Ability to test business hypotheses ● Test strategy evolving with the system
- 4. unit testing integration testing internal service testing black-box service testing functional testing production testing ???????????? $$$$$$$$$$ $ $$$$$$$ $$$$$ $$$ units of code cohesive groups of units core service whole service system of services production
- 5. There are trade-offs ● Execution time ● Impact on release process ● Cost of maintenance ● Execution frequency ● Impact on design ● Achievable coverage
- 6. Lets zoom in
- 7. unit_3 mock unit_1 unit_2 mock Unit test Interacts with a unit of code Unit = class, function, actor, etc. Full isolation, all dependencies mocked No IO operations, no network, no server test_1
- 8. Benefits ● Drive the design ● Code as documentation ● Formal proof ● Lightning-speed execution Trade-offs ● Unit testing legacy codebase ● Brittleness
- 9. Now zoom out
- 11. unit_3 mock unit_1 unit_2 Integration test Interacts with system of few units Testing cohesive units, mock external units No IO operations, no network, no server Testing interactions between units test_1
- 12. The Problem How to decide which units to cover with integration test?
- 14. Good candidates for integration test? ● units from the same package ● units share state between each other ● units lifecycles are coupled ● units experience temporal coupling (order of calls matter) ● units look like one unit from the outside ● integration test does not require much mocking
- 15. Benefits ● Avoid ‘Zero integration tests’ fail ● Code as documentation ● Formal proof Trade-offs ● Should we test behavior or state ● Defining cohesive units is hard ● Where to draw a boundary? ● DB? Filesystem? Concurrency? ● Medium maintenance cost
- 16. Mutation test unit_1test_1 unit_1test_1 unit_1test_1 inject defect inject defect passes fails passes ? bad good
- 17. Benefits ● assures quality of tests ● expresses REAL coverage Trade-offs ● language specific tooling ● adds execution time
- 18. Property based test unit_1test_1 Regular Unit test Property Based Test pre-defined input unit_1test_1 auto-generated input . . . repeat N times
- 19. Benefits ● generalized and randomized input ● discovering edge cases ● improved coverage ● can execute hundreds of tests with one definition Trade-offs ● language specific tooling ● adds execution time
- 21. Benefits ● reduces bugs ● better code design ● detects security issues ● helps assure standards Trade-offs ● hard to cover legacy systems ● adds execution time
- 22. Zooming out even more
- 23. Service ● A graph of code units and their interactions forms a service ● A service is one deployment entity ● The particular arrangement of units in a service is its architecture ● An architecture should be chosen that guarantees testability
- 24. Hexagonal architecture ● Core domain does not refer to tools, frameworks and external services ● All external dependencies in core domain are abstracted to ports ● All external dependencies injected as adapters ● Alternative names: Onion Architecture, Ports and Adapters
- 25. u2u3 Core Domain Ports & Adapters External Service Client Message Queue Database p1 a1 u1 a2 a4 p2 p4 a3 p3
- 27. Internal service test in-mem service stub in-mem queue in-mem repository u2u3 Core Domain Ports & Adapters a1 u1 a2 a4 svc_test_1 a3 p1 p2 p4 p3
- 28. Benefits ● relatively fast execution ● no IO operations, no network ● good isolation - parallel execution ● for monolithic service this is a functional test Trade-offs ● runtime impacted by tests ● architecture constraints ● language-specific setup ● does not test production adapter ● cost of maintaining stubs
- 29. Black-box service test external service stub Message queue Database u2u3 Core Domain Ports & Adapters a1 u1 a2 a4 svc_test_1 a3
- 30. Benefits ● language agnostic ● service works like in production Trade-offs ● architecture constraints ● cost of maintaining stubs ● test harness needs to assure isolation ● orchestration of deployment (not a single process anymore) ● resource consuming
- 31. Keep zooming out
- 33. System ● A system consists of multiple services communicating with each other ● Each service delivers one or more capabilities (business/technology) ● Boundary services are the ones talking to external services and clients ● It should be possible to replace services with stubs for testing ● Services expose their capabilities following a Contract
- 34. Contract ● Expectations of the service defined by the client ● Each client can have different contract with the same service ● Constraints around schema and behavior ● If well expressed, it helps decoupling testing of client and service ● The service can be stubbed based on contract ● The service can be tested based on contract
- 35. ext1contract 1 contract 2 contract 3 svc1 svc1 svc2 ext2contract 4svc3 ext3contract 5svc3 svc3 svc3 Contract test
- 36. Benefits ● specification of entire system ● teams can work independently ● assuring compatibility ● consumer driven contracts Trade-offs ● constraints on process ● maintenance cost ● needs proper tooling
- 38. Benefits ● Tests the system as a whole ● Validates end-user interaction Trade-offs ● High maintenance ● Resource consumption ● How to assure proper state ● Long execution time ● Hard to measure coverage ● Brittleness ● How to spin 20 services up
- 39. if the system has poorly defined boundaries or complex operation modes it is hard to have meaningful tests of isolated services
- 41. Benefits ● Discovering subtle bugs ● Testing user experience ● Source of relevant functional tests Trade-offs ● Manual ● Record-replay is difficult ● Need advanced tooling ● Need experienced test engineers
- 42. Testing in production ● It is possible to perform non-invasive tests on production environment ● A/B testing: exposing different versions of the capability ● Canary release: exposing change to a limited group of early adopters ● Smoke tests: automated testing of few critical path scenarios
- 43. Production rollout slider dark launch canary release A/B testing sunsetting v2 in production, but no traffic directs to it only beta testers and internal employees can access v2 pre-defined populations access different versions cut-over only legacy clients can access v1 new clients only access v2 all clients access v2
- 45. Benefits ● reduces release risk ● helps achieve continuous delivery ● can be fully automated Trade-offs ● requires proper infrastructure ● requires feature flags system ● needs to be built into the system
- 47. Benefits ● validates business hypothesis Trade-offs ● requires proper infrastructure ● requires feature flags system ● needs to be built into the system
- 48. Behavior Driven Development a.k.a. Acceptance Test Driven Development
- 49. Behavior Driven Development: Before Set up logged in user, who never applied for Card and has credit score below 680. When that user applies for Card, their application is accepted. T R A N S L A T I O N
- 50. Behavior Driven Development: After GIVEN the user is logged in AND the user never applied for Card AND the user has credit score > 680 WHEN the user applies for Card THEN the application is accepted T R A N S L A T I O N
- 51. Benefits ● drives the design ● specification is testable ● business understands tests ● highly reusable test code ● living documentation ● facilitates collaboration Trade-offs ● requires expertise ● takes time and skills to set up
- 54. Benefits ● determines performance of small units of code ● drives design of highly performant architectures Trade-offs ● long execution times if the test suite is large ● requires understanding of the runtime in order to set up correctly
- 56. Benefits ● can simulate production traffic ● can validate performance requirements Trade-offs ● expensive set up ● long execution time
- 58. Benefits ● identifies breaking point of the system Trade-offs ● resource intensive ● expensive setup
- 59. Soak (Endurance) test load generator 2 system under test load generator 1 load generator 3 orchestrator observations
- 60. Benefits ● validates stability of the system ● detects resource leaks ● detects difficult bugs Trade-offs ● very long execution time
- 62. Benefits ● proves system resiliency Trade-offs ● expensive set up ● requires well designed system
- 65. Benefits ● ensuring standards are met ● automation ● good coverage ● detects missing patches ● detects common vulnerabilities (e.g. Heartbleed, Shellshock) Trade-offs ● no tests beyond standards ● execution time ● difficult setup, especially in distributed architectures like microservices
- 66. Penetration Testing system under test skilled security engineer
- 67. Benefits ● unique vulnerabilities detection ● non standard testing ● discovering true impact of vulnerabilities Trade-offs ● lots of manual exploration ● no automation ● expert knowledge required ● unclear coverage
- 68. So now you... ● Know types of software testing ● Can decide which tests you need ● Understand and accept trade-offs but there is a lot more to learn
- 69. Questions?
Editor's Notes
- #40: notice the parallel to integration tests