OSMC 2015: Testing in Production by Devdas Bhagat
0 likes287 views
The document discusses various aspects of software testing and development, emphasizing the importance of asking the right questions, understanding metrics for good software, and effective risk management strategies. It outlines the feedback loops in testing environments and the significance of real-time event processing for monitoring software performance. Additionally, it highlights the typical lifecycle of code and the necessity for organizations to adapt to changes and failures in their software systems.
OSMC 2015: Testing in Production by Devdas Bhagat
- 1. I have a test network. You may know it as production. – Andreas Thienemann Testing
- 2. Asking the right questions ● Information is not a scarce resource, attention is – Herb Simon
- 3. Asking the right questions ● Information is not a scarce resource, attention is – Herb Simon ● What do you know?
- 4. Asking the right questions ● Information is not a scarce resource, attention is – Herb Simon ● What do you know? ● What do you not know?
- 5. Asking the right questions ● Information is not a scarce resource, attention is – Herb Simon ● What do you know? ● What do you not know? ● What do you not know that you do not know?
- 7. Fast Feedback Loops ● Test driven business
- 8. Fast Feedback Loops ● Test driven business – It's like Test Driven Design
- 9. Fast Feedback Loops ● Test driven business – It's like Test Driven Design ● Enable IT to speak the language of business
- 10. Fast Feedback Loops ● Test driven business – It's like Test Driven Design ● Enable IT to speak the language of business – Show me the data!
- 11. Fast Feedback Loops ● Test driven business – It's like Test Driven Design ● Enable IT to speak the language of business – Show me the data! – HIPPO
- 12. What makes software “good”?
- 13. What makes software “good”? ● Developer metrics
- 14. What makes software “good”? ● Developer metrics – Object Oriented/Functional/ – Is testable – Follows DRY – ...
- 15. What makes software “good”? ● Developer metrics – Object Oriented/Functional/ – Is testable – Follows DRY – … ● Operational metrics – Bug “free” – Scalable – Secure
- 16. What makes software “good”? ● Business metrics
- 17. What makes software “good”? ● Business metrics – Does it help me make money?
- 18. What makes software “good”? ● Business metrics – Does it help me make money? – Does it save me money?
- 19. What makes software “good”? ● Business metrics – Does it help me make money? – Does it save me money? – Does it generate value?
- 21. Classical SDLC feedback loop
- 22. Plans vs reality
- 24. Testing vs Reality ● Stable environment ● Highly unstable
- 25. Testing vs Reality ● Stable environment ● No humans ● Highly unstable ● Humans
- 26. Testing vs Reality ● Stable environment ● No humans ● Low latency ● Highly unstable ● Humans ● Potentially high latency
- 27. Testing vs Reality ● Stable environment ● No humans ● Low latency ● Not always the same size of dataset ● Highly unstable ● Humans ● Potentially high latency ● Large, ever changing datasets
- 28. Users ● Humans do strange things
- 29. Users ● Humans do strange things ● Or sometimes make mistakes
- 30. Users ● Humans do strange things ● Or sometimes make mistakes ● They come up with different requirements
- 31. Users ● Humans do strange things ● Or sometimes make mistakes ● They come up with different requirements ● They change the world your software works in
- 32. Risk management ● Approach 1: – Scope your problems well – Test a lot – Release stable code – Avoid changing a working system
- 33. Risk management ● Approach 2: – Accept that you have an ill-defined problem – Iterate rapidly – Make a large number of small changes – Build software to be able to isolate these changes – Test them in the real world – Keep only what works
- 34. What if When it breaks? ● Fix fast (maybe) – “Do it right the first time” does not apply ● Business process for handling failure – Hardware will eventually fail, software will eventually work
- 35. The lifetime of code ● How long does your code live?
- 36. The lifetime of code ● How long does your code live? – Hours or days? ● This should be most code out there
- 37. The lifetime of code ● How long does your code live? – Hours or days? ● This should be most code out there – Months? ● A little code, often “libraries” with a single application
- 38. The lifetime of code ● How long does your code live? – Hours or days? ● This should be most code out there – Months? ● A little code, often “libraries” with a single application – Years? ● Very little. Just core libraries
- 39. The lifetime of code ● How long does your code live? – Hours or days? ● This should be most code out there – Months? ● A little code, often “libraries” with a single application – Years? ● Very little. Just core libraries ● It is safe to delete code, if you are using version control
- 40. Event processing ● Generate information about software use and changes in realtime ● For more information and tooling: – https://www.quora.com/Are-there-any-open-source- CEP-tools?share=1 – https://en.wikipedia.org/wiki/Complex_event_proces sing
- 41. Event processing ● Generate information about software use and changes in realtime ● For more information and tooling: – https://www.quora.com/Are-there-any-open-source- CEP-tools?share=1 – https://en.wikipedia.org/wiki/Complex_event_proces sing
- 42. Monitoring/Alerting ● Process events to generate graphs
- 43. Monitoring/Alerting ● Process events to generate graphs ● Riemann is an excellent tool for generating alerts from event streams
- 44. Monitoring/Alerting ● Process events to generate graphs ● Riemann is an excellent tool for generating alerts from event streams ● Generate graphs as close to realtime as possible – Developers doing rollouts know that something else is changing
- 45. Monitoring/Alerting ● Process events to generate graphs ● Riemann is an excellent tool for generating alerts from event streams ● Generate graphs as close to realtime as possible – Developers doing rollouts know that something else is changing – Any major problems will be caught really quickly
- 46. Monitoring/Alerting ● Process events to generate graphs ● Riemann is an excellent tool for generating alerts from event streams ● Generate graphs as close to realtime as possible – Developers doing rollouts know that something else is changing – Any major problems will be caught really quickly ● Isolation of changes means that you can track longer term effects of each change