Grokking Techtalk: Problem solving for sw engineers
- 1. Filling in the gap between realities A guide to problem solving for software engineers Grokking -Thanh Dinh 26/02/2022
- 2. Agenda What is a problem Problem definition Causal analysis Trade-off 2 types of trade-off Separation principles Conclusion
- 3. What is a problem? A tale of two realities
- 4. The elevator problem Lengthy elevator wait People in an office building started to complain to building management about the long wait time for elevators. As part of office building staff, you are tasked with solving this problem. How do you solve it?
- 5. Elevator problem - Quick solutions
- 6. Who has the problem? - The stakeholders
- 7. Stakeholders - Clients vs Users
- 8. What is a problem?
- 9. Perceived vs Expected Realities
- 10. Move perceived reality to expected reality
- 11. Move expected reality to perceived reality
- 12. Pull 2 realities to a middle point
- 13. Fill in the gap of residents' realities
- 14. Fill in the gap of Holistics customers' realities
- 15. Problem definition Choosing the right problem statement to solve
- 16. A problem well-stated is a problem half-solved. John Dewey
- 17. Startup is about finding the right problem to solve. Anonymous
- 18. Problem definition What is the problem? What is perceived reality? What is expected reality? Is it clear? Who are involved in the problem? How important is the problem? When does the problem needs to be solved? Is there really a problem?
- 20. Highest abstraction level Problem definition should be at the highest abstraction level that appropriately describes the situation. Principle
- 21. The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him. Miyamoto Musashi
- 22. Example: Holistics Analytics as Code Lower level: product features Version control Logic as code Automated workflow Data unit testing Higher level: pain points Can't track who changed what Unreliable dashboards Manual, repetitive work Vendor lock-in Why How Lowest level: technology choice Analytics Modeling Language (AML) TypeScript Redis Postgres Terraform/Ansible Why How
- 23. Ladder of abstraction: Product Design
- 24. Causal analysis Knowing what causes the problem
- 25. The elevator problem Lengthy elevator wait People in an office building started to complain to building management about the long wait time for elevators. As part of office building staff, you are tasked with solving this problem. How do you solve it?
- 26. Causal diagram
- 27. Non-linear causal relationship In many cases, the causal chain is not linear that can be described as a tree. Changing one variable can affect itself through cycles. Cycles mean there's no definite root cause of the problem. Pitfall
- 29. Non-holistic thinking Company growth problem is not just a sales, marketing, engineering, product problem, but the whole company problem. Usually, a problem is caused by interactions between components or the compound effect of multiple dimensions, not the individual parts/dimensions. There's no single root cause to find. Pitfall
- 31. Forget that problem is inherently a human thing Definition of a problem requires a person to even be called a problem but many engineers forget this. Forget to choose the right problem to solve Engineers tend to think that "requirements" are set in stone. In reality, most requirements may not even make sense in the first place. Common mistakes
- 32. Trade-off Which one will you choose?
- 33. You face a trade-off when it is hard to choose a solution Thus you must have already found solutions in order to have trade-offs
- 35. Causal Analysis and Trade-offs
- 36. Dimensions
- 37. In SE, dimensions come from business requirements
- 38. Dimensional priorities Talk to stakeholders to identify the most important dimensions
- 39. There are 2 types of trade-off
- 40. Dimensional trade-off A solution increases benefit in one dimension but also increases harmful effect in another dimension Alternative trade-off Solution 1 gives you benefit 1, solution 2 gives you benefit 2, you can only choose either, but you want both
- 41. Dimensional trade-off examples More security checks will improve security but reduce performance 🔐Security vs 🐎Performance More tests will improve code reliability but it will take longer to ship new features ⛑Reliability vs 🔀Agility Migrate to Kubernetes improves scalability but increases infrastructure cost 🚀Scalability vs 💸Cost It is hard to design a software that is both powerful and user-friendly 💖User- friendliness vs ⚡Power
- 42. Alternative trade-off examples SQL allows for complex joins but require upfront schema design SQL vs NoSQL Monolithic architecture is simpler but less scalable Monolithic vs Microservices Fundamental trade-off between functional programming and OOP Expression problem
- 44. Choose a compromise point Find a sweet spot on the line of compromise Resolve the trade-off Use creative problem solving techniques to overcome the trade-off 2 ways to deal with trade-offs
- 45. Only see one side of the trade-off Only seeing one side of the trade-off means not seeing the trade-off at all! Common mistake 1
- 46. Programmers know the benefits of everything and the trade-offs of nothing. Architects need to understand both. Rich Hickey
- 47. Everything in software architecture is a trade- off. If an architect thinks they have discovered something that isn’t a trade-off, more likely they just haven’t identified the trade-off yet. Neal Ford
- 48. Opinion: Use boring technology New New benefits Unknown harms New/fancy tech Old Old benefits Known harms Boring tech http://boringtechnology.club/
- 49. The nice thing about boringness (so constrained) is that the capabilities of these things are well understood. But more importantly, their failure modes are well understood Dan McKinley
- 50. Not realize that trade-offs are not set in stone Trade-offs are hard to overcome but in many cases, they are not set in stone. You can resolve them through separation techniques. Common mistake 2
- 51. The root of many trade-offs are due to couplings. Resolve the coupling, you can resolve the trade-off Why separation?
- 52. TRIZ separation principles applied to Software Engineering Break down different requirements into separate time group Separation in Time Break down different requirements into separate modules Separation in Space Break down different requirements at the submodule and system level Separation between Parts and Whole
- 53. Separation in Time Agility/Cost vs Reliability/Security/Scalability
- 54. Separation in Space Agility: Separate service can be developed independently Scalability: New service can be connected to the common sidecar Security/Reliability: Handled by sidecar Performance: Communication overhead Cost: Only makes sense for more complex systems Security/Reliability vs Agility/Scalability Sidecar pattern
- 55. Separation between Parts and Whole Agility/Scalability vs Reliability Agility: Fast many unit tests without mocks Scalability: Functional logic can be easily parallelized Reliability: Only need a few reliable integration tests, most logic covered by unit tests Performance: Communication/serialization overhead Cost/Learnability: More time decoupling stateless from stateful logic Functional core Imperative shell
- 56. Conclusion What is a problem A problem is a gap between perceived and expected reality of some stakeholder Look at a problem at its highest level of abstraction: its ultimate goal Causal analysis Trade-off Solution ideality and relationship with causal analysis 2 types of trade-off: dimensional/alternative 3 separation techniques to resolve trade-offs
- 57. Further readings
- 58. Bringing software's best practices to analytics development Old World New World Software Development Analytics Configure servers manually using GUI. Deploy via FTP upload. Problems: Hard to maintain; Prone to error Configure servers using code (Infrastructure As-Code). Continuous Integration / Continuous Delivery (CI/CD) Benefits: Reliable, Scalable and Maintainable. Build visualizations using drag- and-drop. Hard to reuse business logic. Analytics built with software principles Holistics
- 59. 1- Define analytics logic as code 2- Code review, deploy with CI/CD 3- Reliable, trusted analytics
- 61. Our Culture Problem-Solving Centric: (as seen from today's talk) We think from first principles, question assumptions, and deliberate tradeoffs carefully. Personal Development: We invest so that as the company grows, the people grow. Technical Coaching & Mentorship Pair Programming Learning by Teaching...
- 62. Building a language for analytics modeling Example Technical Problem Powerful type system to enable rich IDE functionalities A query language that is composable/reusable alternative to SQL Goal: Making analytics reusable Solution: A type-based DSL layer that allows mapping between business logic to database layer. Technical challenges: