Distributed systems: design, principles and experiencies
- 2. Who am I? • Senior Software Engineering and Data Engineer • More than 15++ years of experience • Had fun with tons of projects with different industries. • …and I’ve started my career working as a game developer
- 5. • We use apps and services every day. • For us, this is software • We don’t care where they are. • We don’t care where we are. • We don’t care when we are using it • Yep, anytime • We don’t care which device we use. • They just work • For us, let’s say they are somehow distributed
- 7. Let’s think a bit
- 9. How we do tests?
- 12. Quick review on minimal practices • SOLID: • S - Single Responsibility Principle: A class should have one reason to change. • O - Open/Closed Principle: Software entities should be open for extension but closed for modification. • L - Liskov Substitution Principle: Subtypes must be substitutable for their base types. • I - Interface Segregation Principle: Clients should not be forced to depend on interfaces they do not use. • D - Dependency Inversion Principle: High-level modules should not depend on low-level modules.
- 13. Any software should solve a problem • And we use code for that. How? • Architecture. • Good practices. • Testing • Design patterns • Monitoring • Standards • And there is a name for this: software engineering
- 16. Let’s talk about architecture
- 17. Let’s talk about architecture
- 19. And if I’m building a distributed app, will be easier, right? Right?
- 20. Why make an app distributed? • Redundancy • High availability • Geographical optimization • Law restrictions • Performance • …
- 21. How? • Deploying same app in different places
- 22. How? • How to scale data? • Multiple databases • Sharding • etc
- 23. How? Moving away our monolith • Moving to services • Split the app in smaller pieces • Bounded contexts • Gateways • Circuit-breaker • …and more other patterns
- 24. How? Moving away our monolith • Using a standard communication • REST APIs • GraphQL • gRPC • Using structures for decoupling • Publisher / Subscriber • Queues • BFF • Webhooks
- 25. Example: REST endpoints • GET /v1/users • GET /v1/users/12345 • POST /v1/users • PUT /v1/users/12345 • DELETE /v1/users/12345
- 27. Most common HTTP status codes • 200 • 201 • 202 • 400 • 403 • 404 • 429 • 5xx
- 29. Microservices • Autonomous • Independent • Own data system, independent • If it shares database with other thing it could be fine, but it is NOT a microservice • Exposed via standards (HTTP, etc)
- 31. Netflix example
- 32. Microservices • Big scaled orgs • Should be autonomous at all levels • Cultural fit • It should represent the solution of a scale problem, which can be technical or human
- 34. Tips and tricks: decision making • Ask yourself this: why do you need to have the app distributed? • If the answer is “no idea”, don’t waste time now. • If the answer is “someone told me that Netflix does it” let’s go for a long walking • If there is a reason, then: • Redundancy/high availability? Go simple • Performance? Divide and conquer • Laws? Do the minimum
- 35. Tips and tricks: decision making • Early optimizations without a reason are expensive • Early architectural decisions without a reason is a pain • There is always time to scale and change the things. Market will tell you. • But keep in mind: you have to measure the impact of your decision. We do science, not magic.
- 36. Tips and tricks: cultural fit • Is your company prepared to communicate only with your services? • Are you able to integrate/build software against services you don’t know? • Can you keep your service updated? • Can your service be down?
- 37. Tips and tricks: API and communications • Expose a CLEAR API following standards: • Use OAuth for authentication • Expose endpoints you will support: • Use versioning • Follow standards • Define HTTP verbs used • Define rate limiting • Define headers • Prepare a docs explaining your API with examples. • Ensure your doc can be updated at same time your API does • Include best practices, how you want your API is used, etc. • Most important: you and your team should be the first user of your API
- 38. Tips and tricks: Deployment • Define how your app/service/microservice can be deployed • The infra should be coded (Cheff, Terraform, CLI, whatever) • Any deployment should be automated • Can your deployment break other parts? • Any deployment should support to do a rollback • Define a strategy for rollback / troubleshooting • Document new scenarios / cases • Prepare backups if data needs to be changed • In most cases, you can live without your app running, but not without the customers data
- 39. Tips and tricks: Monitoring • Use tools for monitoring your app: Log analytics, Cloud Watch, Datadog, etc • Define what is an alert and what to do: • Create a ticket for handle it • Define on-call strategies • Must be able to know what is happening in your app • And yes, if you want to track multiple services, it is more complicated. • In general, be reactive and not passive
- 40. Tips and tricks: Scaling • Analyze and measure what is the scaling need: • In terms of resources: CPU, disk, memory, bandwidth • In terms of services: login, catalog, jobs, etc. • If you have a problem scaling could be a quick solution to sort the issue, not the real solution. • Decide what kind and when to scale: horizontal, vertical, etc. • Add rules to handle scaling: by schedule or by metrics
- 41. Tips and tricks: general • There is no silver bullet for doing a distributed apps • But there are few good practices that are recommended to follow • This is a continuous process, not an end destination. Time will tell you your needs, problems and limitations.
- 44. Distributed Systems render farm tutorial • Go to https://github.com/khnumdev/dist-app-tutorial and follow the steps
Editor's Notes
- #2: Remind events and sessions, including UA until this year :/
- #3: Home computers without internet: only stuff you have installed Dotcom bubble Q: what was the first company to be considerd as SaaS? Amazon case with AWS and rest APIs
- #4: 2007: iphone release 2010-2015: “transformacion digital” 2018-2022: blockchain and bitcoin >2022: IA